User interface system for movement skill analysis and skill augmentation

ABSTRACT

A cue processor uses one or more sensors to obtain motion data for a user performing a physical task in an environment. A cueing law is based on a model determined from the motion data, for example a movement and skill model where the collected motion data are parsed into one or more movement units used to accomplish a range of outcomes. The cue processor generates a movement phase estimation to predict a movement phase and associated movement feature, and applies the cueing law to generate a cue signal. The cue signal is communicated to the user as a visual, audio or haptic stimulus, selected to target the feature for the user to achieve or improve a desired outcome.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.17/068,012, filed Oct. 12, 2020, entitled SYSTEMS AND METHODS FORMOVEMENT SKILL ANALYSIS AND SKILL AUGMENTATION, which is a continuationof U.S. patent Application No. 15,247,622, filed Aug. 25, 2016, entitledSYSTEM FOR MOVEMENT SKILL ANALYSIS AND SKILL AUGMENTATION AND CUEING,issued Dec. 1, 2021 as U.S. Pat. No. 10,854,104, which claims thebenefit of U.S. Provisional Application No. 62/211,281, filed Aug. 28,2015, entitled PLATFORM SYSTEM FOR MOVEMENT SKILL ANALYSIS AND SKILLAUGMENTATION AND CUEING, each of which is incorporated herein byreference, in the entirety and for all purposes.

BACKGROUND

Disclosed are devices, systems and methods for movement skill analysis,movement skill augmentation and movement cueing using decomposedmovement elements for modeling and analysis.

Humans rely on motion skills to perform daily tasks ranging from actionsessential to our autonomy to more specialized domains requiring highlyrefined motion skills. Professional athletes, musicians, surgeons, andeven elite amateurs require thousands of hours of systematical andfocused training, as well as continued training to maintain high skilllevels. Even simple daily acts involve complex coordination of a rangeof processes, from sensing and motor-control to perception andcognition. Learning, maintaining, and rehabilitating movement skills arevaluable, but at the same time, complex and challenging tasks. Acquiringand maintaining specialized movement skills takes time. Progress ofmovement skills does not develop linearly with training time. Rather,skills progress exponentially with what is left to learn in order to beproficient at the movement.

Movement performance relies on a broad range of functions (e.g.,sensory, perceptual, planning, cognition). Many movement skills withinthe category of complex movement are unnatural and therefore requireadaptation of innate functions to the specific task requirement andfamiliarization with the external task elements.

Complex movement requires coordinating large numbers of muscles and bodysegments. The computational requirements for movement need to beinitiated within short time-frames and often need to be adapted duringperformance.

Movement is typically learned by trial and error, focusing on theoutcome. The specific details regarding how movement is organized areonly known implicitly in procedural memory. Explicit knowledgesurrounding movement details are typically not used during practice andexecution. Movements often unfold too quickly to even be perceived.Complex movements involve too many dimensions. For example, the path ofa piece of equipment, such as a tennis racket, involves combining threetranslational and rotational variables (e.g., six degrees of freedom).

This complexity grows exponentially when various body segments andbiomechanical and neuro-motor constraints are included. To make matterseven more complex, these variables are constrained by the dynamics,which constrain their spatial and temporal evolution. Finally, there arevery few feedback stimuli, or signals, available to a user during thetraining process. As a result, for most people who don't have access tocoaching, movement skill relies on self-observation and tediousrepetition. In many domains, proficiency cannot be achieved without theassistance of an expert coach or trainer.

Challenges exist in characterizing and assessing movement. First, humanmovement is variable. Each repeated trial of the same task results in aslightly different execution. Second, technique is idiosyncratic.Individuals with the same general level of ability have a differentapproach and style. Third, movement is fast. Often an action unfoldswithin a fraction of a second, with relevant details only spanning a fewmilliseconds. Fourth, movement is complex. It often focuses on thecontrol of an end effector, such as a tool (e.g. surgical instrument) orpiece of equipment (e.g., tennis racket, baseball bat, golf club), whichneed to be controlled in three dimensional workspaces. The execution ofsuch movements, however, requires controlling the various limbs, joints,and muscles, which add many more additional degrees of freedom.

Moreover, coordinated motion patterns are typically too complex andexecute too quickly to be perceived and processed consciously.Therefore, it is usually impossible to make training interventions inreal time. Moreover, athletes or operators usually don't have asufficiently explicit awareness of the details of their motionexecution. These characteristics explain why it difficult to improveskills once basic motion patterns are acquired. External feedback from atrainer or coach becomes necessary in order to improve.

Movement skills also depend on a perceptual understanding of theexternal task elements. These characteristics are much harder to assessfrom observations of the movement performance. They manifest indirectlyin the performance. A good instructor will call his attention toimportant perceptual cues and response characteristics and givediagnostic knowledge of results.

Finally, an important requirement for effective training is to accountfor individual differences in body type, skill level, health, etc. Suchcharacteristics are much harder to explicitly take into account duringtraining. It is particularly critical for rehabilitation or when workingwith injured or aging athletes. A training approach should also leveragethe properties and natural learning principles and processes of skilldevelopment.

Popular wearable and embedded devices currently primarily focus on theidentification and tracking of activity (e.g. Fitbit® fitness trackeravailable from Fitbit, Inc. or Jawbone® fitness tracker available fromAliphcom doing business as Jawbone). Popular examples of fitnesstrackers include devices for counting steps and tracking distancecovered. More advanced capabilities can be found in devices that arespecialized for a particular sport. Tennis, badminton, and golfrepresent the largest market segments (see e.g. BABOLAT® from Babolat, aFrench company), ZEPP® tennis swing analyzer (available from Zepp US,Inc.), and the SMART TENNIS SENSOR available from Sony). These productsaim to provide an analysis of players' technical performance. Typicalfeatures include tracking the type of actions; reconstructing movements,such as the path of the tennis racket during a stroke; tracking keyoutcome variables of actions such as the racket head speed, thedistribution of impacts on the sting bed, and the amount of spin.

The outputs of these assessments are typically provided after a trainingor play session. The data is presented as summaries of sessionperformance, as well as time. The data is also aggregated to providestatistical trends. The main shortcoming of these products is that theanalysis is based on outcome variables (referred to as knowledge ofresults in the human skill literature) and thus does not provideactionable information that can be leveraged directly for training.

What is needed are systems and methods that provide real-time andpost-performance training and rehabilitation to a user which providesaugmentation of human movement and takes into consideration variouscomponents of the movement.

SUMMARY

A human movement augmentation system is disclosed that provides variousforms of feedback, including real-time feedback and post-performancefeedback for training and rehabilitation. The devices, systems andmethods are configurable to augment human movement behavior in order toaccelerate complex movement skill acquisition, improve outcome andperformance, and mitigate injuries and wear from poor or maladaptivemovement technique.

A difficulty with skilled human movement behavior is that the dataneeded to describe the performance is high-dimensional and includesother complexities such as nonlinearities arising from movementdynamics. Moreover, human performance is highly variable, even for asingle individual. In addition, humans present a broad range of skilllevels, styles, body types, ages, physical conditions, and medicalconditions. All of these variables affect movement performance. Themovement complexity has long eluded researchers who study the humannervous system (see e.g., Bernstein's degree of freedom problem).

A method is disclosed that decomposes movement into elements that followthe natural structure and organization of the human nervous system.These elements are used to model and analyze movement behaviorsystematically, and subsequently can be used to synthesize and implementvarious forms of movement augmentation to improve skills consistent withnatural human movement control principles.

Additionally, an environment that enables systematic and comprehensiveimprovement of capabilities through the integration of its components.On the input side, the invention assumes a method for collectingmovement measurement data of the equipment and various body segments inrelationship to task and environmental elements that are relevant to thelevel of analysis considered in that activity. The measurementsencompass data from wearable sensors, such as MEMS IMUs, and/or datafrom technologies including a vision or optical based tracking system.On the output side, the invention uses various modalities to communicatefeedback, including visual, haptic, and audio; these are embedded eitherin pieces of equipment or in accessories (smart watches, smart glasses,etc.).

The components of the movement skill augmentation system include: AMovement Processing System (MPS) to decompose movement data according tothe hierarchical movement system organization.

The first step extracts movement segments that correspond to movementprofiles of the primary movement units (PMU) used in performing a task.The second step of MPS delineates movement data into movement categoriesand classes according to movement intrinsic characteristics,interactions with the task environment, and/or the movement's outcomes.

The collection of PMU defines the subject movement repertoire. The PMUmovement profiles from each class are then segmented into phases.Finally, the phase profiles are decomposed into movement components thatcan be related to the muscular activation patterns and joint motionsunderlying the movement coordination.

The Motion Model (MM) captures movement behavior following the threelevels of organization of the human movement system used for the MPSprocessing stages. The repertoire of PMU formed by aggregating thecategories or classes describes how a subject partitions the space ofoutcomes required to perform a task into distinct families of movementpatterns with their associated outcomes.

The phase-segmented PMUs are used to define a finite-state model thatdescribes movement pattern as a sequence of states with specificdynamics. The states correspond to the individual movement phases, whichare driven by the underlying continuous movement process.State-transition conditions determine the sequencing of the phases. Thismodel describes how complex movement patterns are achieved in the faceof musculoskeletal and other biological constraints. Finally, theindividual movement components, obtained by decomposing the phaseprofiles, are mapped to the musculoskeletal system to describe how themovement profiles are achieved by combining body segments and musclesynergies.

The Skill Model (SM) extracts attributes from the motion model. Theskill model is divided into two levels. The task performance assessmentlevel describes the range of outcomes and reliability of the movement inachieving these outcomes. The functional assessment level describes howthe subject uses movement technique to achieve various outcomes, and howmovement technique is adapted to compensate for variations anduncertainties in task conditions and movement execution.

The movement phase elements provide a sparse description of thehigh-dimensional movement data. Since it is grounded in human structure,it plays a central role in functional analysis. The elements of themovement model, including phase transition configuration, phase profilecharacteristics, etc. are used to identify what features are mostpredictive of the outcome.

Similarly, movement adaptation mechanisms can be analyzed by studyingpatterns of coupling between movement component features that have noeffect on the outcome. The skill model is used to determine theindividual's Skill Status (SS), which provides a comprehensive skillassessment. At the activity performance level, it encompassesrepertoire, statistical characteristics of movement pattern profiles,and associated outcomes. At the functional level, it encompasses theoutcome feature sets and adaptation feature sets for each movementpattern and their associated movement model. Finally, the biologicaldetails include muscle and joint coordination and performance, which aredescribed by the movement synergies and biomechanical models.

A method for the synthesis of feedback augmentation. The augmentationsare divided into Instructions, which operate at the knowledge level, andFeedback Cues, which operate at the signal and cue level of the humaninformation processing and skill hierarchy. The Instructions operate atthe knowledge level and are implemented through various visualizationmethods that describe the elements of the motion and skill models suchas maps of the repertoire, graphs of the movement profiles for differentmovement patterns, and detailed graphs of the phase-segmented movementprofiles, and at the lowest level, the movement synergies and associatedmovement biomechanics.

The visual instructions are also accompanied by verbal instructions thatexplain the relevant quantities. At the functional level, theinstructions detail the role of the elements of the movement model ingenerating outcomes, including the features related to movementconfiguration at phase transitions, and profile features of the variousphase segments. The synthesis of Feedback Cues is based on CueingMechanisms that are derived from the motion model and its variousfunctional features that are related to the natural movementimplementation.

A method and system to realize or implement the different forms offeedback augmentations for training and rehabilitation and injuryprotection. The augmentation is divided between the Instructions and theFeedback Cues. The system components necessary for generatinginstructions are implemented by a host computer with the necessarydisplay and user interface such as a smart phone or a tablet. The systemcomponents necessary for generating feedback cues are implemented by theCueing System. The cueing relies on real-time movement data processing.

The cueing system is divided into two subsystems: The Cue Processor (CP)and the Cue Generator (CG). The CP takes movement data from theavailable measurement systems or devices and generates a Cue Signal. TheCG takes the Cues Signal and generates cue stimuli that can be perceivedby the human individual. The CP is composed of a Phase-state Estimator(PE) and a Cueing Law (CL). The PE estimates the current and futuremovement phases and extracts relevant movement data. The CL takes thephase information and various movement data and computes a cue signal.The CG is composed of a Cue Encoder (CE) and a Cue Transducer (CT). TheCE takes the cue signals and generates a feedback signal that can beinterpreted by the human subject. The CT takes the feedback signal andgenerates physical stimuli (e.g., audio, visual, haptic).

The movement model and augmentation mechanisms provide the componentsfor computer assisted training (CAT). CAT handles various aspects of themanagement and operation of the augmented training process. The CATsystem has two primary subsystems: a Training Agent (TA), which servesas virtual coach that manages the training process, and a Cueing Agent(CA), which controls the Cueing System and manages the various cueingfunctions during play or performance. The Training Agent analyzes thediagnostic results to formulate a Training Schedule (TS). At therepertoire level, the TA determines which movement patterns must beoptimized or refined, and also specifies what new movement patterns mustbe formed. The former improves the skills of specific movement patternsand the latter is directed at broadening the repertoire either throughthe introduction of a new pattern or by helping the differentiation ofmovement within an existing pattern.

The Training Schedule is encoded as a sequence of Training Elements(TE). These are either target motion pattern refinement or movementpattern formation. The TE are formalized as a hierarchical tree thatspans the various Training Paths. The information about the hierarchicalrelationships between the patterns in the repertoire and the relativeimportance of the various patterns for the activity domain is used todefine the ordering and timing of the TE. The Training Agent tracks theSkill Status and the progress along the training schedule and updatesthe TEs. The longitudinal changes in Skill Status corresponds to theindividual's learning curve.

The Cueing Agent manages the cueing process for a particular TrainingElement of the TS. The TEs typically consist of pattern refinement orpattern formation. The Feedback Cue that can be activated to augmenttraining of the Training Element consists of a set of Cueing Elementsthat constitute the Cueing Profile (CPr). Each Cueing Element ischaracterized by a Cueing Mechanism and is realized by the Cueing Systemthrough a cueing law. The Cueing Agent tracks the effectiveness of theCueing Profile and the individual Cueing Elements.

In addition to these capabilities, the data structure associated withthe motion and skill models provide features to support efficientmanagement, organization and communication of movement data. Thequantities that are described above can be managed for variouspopulations of individual along with any relevant parameters such asbody type, size, age, injuries, equipment, etc. The metadata extractedfrom this system can be used to optimize models, algorithms and cueingmechanisms, and provide long-term optimization of training. Theextracted metadata can also improve the understanding of other long-termmovement characteristics such as aging, wear, development of injuries,etc.

Other functionalities and systems related to the overall environmentinclude the Cue Designer. This system provides user interactions at thetraining agent level to optimize training schedule and cueing laws andcue profiles for professionals.

An aspect of the disclosure is directed to a motion analysis system. Themotion analysis system comprises: one or more sensors configured toobtain motion data; a processor in communication with one or moresensors, the processor configured to: collect a motion data from the oneor more sensors; parse the collected motion data into one or moremovement units; model the one or more movement units; analyze the one ormore movement units; compare the one or more movement units to one ormore of each of a prior movement measurement and a library of movementmeasurements to generate a comparison; identify one or more aspects ofthe motion data for change based on the comparison; and provide feedbackto a user. The one or more sensors can obtain motion data with two ormore degrees of freedom that are selected from three axes in Euclideanspace or selected from three measured quantities. Additionally, themotion data obtained can be a movement within a repertoire of usermovements.

Moreover, one or more sensors can be selected from one or more of eachof a wearable sensor and a remote sensor. Wearable sensors can furtherbe configurable to sense one or more of each of a velocity of themotion, an orientation for the motion, a gravitational force, and anelectrical activity of a muscle. Feedback provided to the user can beone or more of each of visible, haptic, and audible. Additionally, oneor more movement features can be computed that impact a quality oroutcome of the feedback. Moreover, the one or more movement features areone of a phase transition or phase profile attribute. The feedbackprovided to the user can be one or more of each of a phase transitioncue, a phase profile cue, an alert, and an outcome validation cue.

Additionally, the feedback can be provided real-time, near real-time, orat a time remote to an actual movement session. Alerts can be computedfrom the map of movement feature to biomechanical features (e.g. toprotect from injury or wear). One or more movement units can be furthersegmented into two or more movement phases. Two or more movement phasescan be decomposed into two or more synergies wherein the two or moresynergies are one or more of each of biomechanical relationships andneuro-muscular relationships. A local host device, such as a localcomputing device, can be configurable to communicate one or more of eachof a training information and a cueing information to the user. One ormore parsed motion data for change can be prioritized based on whether afeature of the movement unit is changeable and will impact a quality oroutcome of a repeated movement. Additionally, the collected motion datacan be one or more of a user motion data and a user controlled devicemotion data.

Another aspect of the disclosure is directed to a means for motionanalysis. The means for motion analysis comprises: one or more sensormeans configured to obtain motion data; a processor means incommunication with one or more sensor means, the processor meansconfigured to: collect a motion data from the one or more sensor means;parse the collected motion data into one or more movement units; modelthe one or more movement units; analyze the one or more movement units;compare the one or more movement units to one or more of a priormovement measurement and a library of movement measurements to generatea comparison; identify one or more aspects of the motion data for changebased on the comparison; and provide feedback to a user.

The one or more sensor means can obtain motion data with two or moredegrees of freedom that are selected from three axes in Euclidean spaceor selected from three measured quantities. Additionally, the motiondata obtained can be a movement within a repertoire of user movements.Moreover, one or more sensor means can be selected from one or more ofeach of a wearable sensor means and a remote sensor means. Wearablesensor means can further be configurable to sense one or more of avelocity of the motion, an orientation for the motion, a gravitationalforce, and an electrical activity of a muscle. Feedback provided to theuser can be one or more of visible, haptic, and audible. Additionally,one or more movement features can be computed that impact a quality oroutcome of the feedback.

Moreover, the one or more movement features are one of a phasetransition or phase profile attribute. The feedback provided to the usercan be one or more of a phase transition cue, a phase profile cue, analert, and an outcome validation cue. Additionally, the feedback can beprovided real-time, near real-time, or at a time remote to an actualmovement session. Alerts can be computed from the map of movementfeature to biomechanical features (e.g. to protect from injury or wear).

One or more movement units can be further segmented into two or moremovement phases. Two or more movement phases can be decomposed into twoor more synergies wherein the two or more synergies are one or more ofeach of biomechanical relationships and neuro-muscular relationships. Alocal host device means, such as a local computing device means, can beconfigurable to communicate one or more of a training information and acueing information to the user. One or more parsed motion data forchange can be prioritized based on whether a feature of the movementunit is changeable and will impact a quality or outcome of a repeatedmovement. Additionally, the collected motion data can be one or more ofa user motion data and a user controlled device motion data.

Still another aspect of the disclosure is directed to a cue processors.Suitable cue processors are configurable to: generate a movement phaseestimation which provides a prediction of a movement phase and anassociated feature, extract a movement phase feature, apply a movementcueing law; and generate a movement cue, wherein the movement phaseestimation and the movement phase feature extraction includes one ormore of each of a phase initiation predictor, an initial phase stateextractor, a phase profile parameter extractor, and an outcomeextractor. The cueing law is further configurable to at least one ofcompare a reference timing, compare a target state value, compare areference profile, and compare a target outcome.

Additionally, the cue generator is configurable to generate one or moreof each of a phase transition cue, a phase profile cue, an alert and anoutcome validation cue. Moreover, the cue processor can be incorporatedinto a stand-alone device. The cue processor can be part of a cueingsystem having a phase state estimator, a cue encoder and a transducer.One or more cues generated can be prioritized based on one or more ofeach of decomposed motion data for change based on whether a movementfeature is changeable and will impact a quality of a repeated movement.

Yet another aspect of the disclosure is directed to a cue processormeans. Suitable cue processor means are configurable to: generate amovement phase estimation which provides a prediction of a movementphase and an associated feature, extract a movement phase feature, applya movement cueing law; and generate a movement cue, wherein the movementphase estimation and the movement phase feature extraction includes oneor more of a phase initiation predictor, an initial phase stateextractor, a phase profile parameter extractor, and an outcomeextractor. The cueing law is further configurable to at least one ofcompare a reference timing, compare a target state value, compare areference profile, and compare a target outcome.

Additionally, the cue generator means is configurable to generate one ormore of a phase transition cue, a phase profile cue, an alert and anoutcome validation cue. Moreover, the cue processor means can beincorporated into a stand-alone device. The cue processor means can bepart of a cueing system means having a phase state estimator, a cueencoder and a transducer. One or more cues generated can be prioritizedbased on one or more of decomposed motion data for change based onwhether a movement feature is changeable and will impact a quality of arepeated movement.

Still another aspect of the disclosure is directed to motion trainingprograms. Suitable motion training programs comprise: one or more sensorinputs configurable to collect motion data; a computerized motiontraining program, wherein the computerized motion training program isconfigured to: parse the collected motion data into one or more movementunits, compare the one or more movement units to one or more of a priormovement measurement and a library of movement measurements to generatea comparison, and present at least one of a training assessment andtraining instruction to the user about one or more of a repertoiremovement, a motion phase segment, and a movement synergy.

Additionally, the motion training program is configurable to provide aplurality of prompts to a user during a sensed motion. In at least someconfigurations, the computerized motion training is iterative. An outputof the motion training program includes a training schedule and one ormore instructions. The training schedule can identify one or morepatterns in a repertoire movement to be practiced by a user. Thecollected motion data can include one or more of a user motion data anda user controlled device motion data.

Another aspect of the disclosure is directed to motion trainingprograms. Suitable motion training programs comprise: one or more sensorinput means configurable to collect motion data; a computerized motiontraining program, wherein the computerized motion training program isconfigured to: parse the collected motion data into one or more movementunits, compare the one or more movement units to one or more of a priormovement measurement and a library of movement measurements to generatea comparison, and present at least one of a training assessment andtraining instruction to the user about one or more of a repertoiremovement, a motion phase segment, and a movement synergy.

Additionally, the motion training program is configurable to provide aplurality of prompts to a user during a sensed motion from a promptingmeans. In at least some configurations, the computerized motion trainingis iterative. An output of the motion training program includes atraining schedule and one or more instructions. The training schedulecan identify one or more patterns in a repertoire movement to bepracticed by a user. The collected motion data can include one or moreof a user motion data and a user controlled device motion data.

A method of training is also disclosed. Suitable method steps oftraining comprise: collecting a motion data from one or more sensors;parsing the collected motion data into one or more movement units;modeling the one or more movement units; analyzing the one or moremovement units; comparing the one or more movement units to one or moreof a prior movement measurement and a library of movement measurements;and identifying one or more aspects of the motion data for change basedon the comparison.

Additionally, the method can include the step of providing feedback tothe user, such as real-time feedback, near real-time feedback orfeedback provided at a later time. In at least some configurations, thefeedback is at least one of an instruction and a cue, such as one ormore of each of visible, haptic, and audible cues. Additionally, themethod can include generating a training schedule based on the one ormore parsed motion data identified for change.

The one or more movement units can further be segmented into two or moremovement phases. Additionally, the two or more movement phases can bedecomposed into two or more synergies wherein the two or more synergiesare one or more of each of biomechanical relationships andneuro-muscular relationships. In some configurations, a local hostdevice is configurable to communicate one or more of each of a traininginformation and a cueing information to the user. The collected motiondata can be one or more of each of a user motion data and a usercontrolled device motion data.

INCORPORATION BY REFERENCE

All publications, patents, and patent applications mentioned in thisspecification are herein incorporated by reference to the same extent asif each individual publication, patent, or patent application wasspecifically and individually indicated to be incorporated by reference.See, for example, US 2013/0053190 A1 published Feb. 28, 2013 for RACKETSPORT INERTIAL SENSOR MOTION TRACKING AND ANALYSIS now U.S. Pat. No.8,944,940 B2 issued Feb. 3, 2015; U.S. Pat. No. 8,602,922 B2 issued Dec.10, 2013, for METHOD AND APPARATUSES FOR ENHANCING PERFORMANCE IN RACKETSPORTS, US 2005/0017454 A1 published Jan. 27, 2005, for INTERACTIVEGAMING SYSTEMS WITH HAPTIC FEEDBACK; US 2007/0105664 A1 published May10, 2007 for RACQUET WITH ENTERTAINMENT AND PERFORMANCE FEEDBACK; U.S.Pat. No. 4,257,594 A issued Mar. 24, 1981 for ELECTRONIC ATHLETICEQUIPMENT; U.S. Pat. No. 8,337,335 B2 issued Dec. 25, 2012, for SYSTEMSAND METHODS FOR MEASURING AND/OR ANALYZING SWING INFORMATION; U.S. Pat.No. 5,646,911 A issued Jul. 8, 1997 for TENNIS PACER; U.S. Pat. No.5,226,650 A issued Jul. 13, 1993 for TENNIS RACKET; US 2002/0077189 A1published Jun. 20, 2002, for PROPRIOCEPTIVE GOLF CLUB WITH ANALYSIS,CORRECTION AND CONTROL CAPABILITIES; U.S. Pat. No. 5,031,909 A issuedJul. 16, 1991 for ELECTRONIC ATHLETIC EQUIPMENT; US 2006/0025229 A1published Feb. 2, 2006, for MOTION TRACKING AND ANALYSIS APPARATUS ANDMETHOD AND SYSTEM IMPLEMENTATIONS THEREOF; U.S. Pat. No. 4,303,241 Aissued Dec. 1, 1981, for SPORTS VISION TRAINING DEVICE; U.S. Pat. No.7,891,666 B2 issued Feb. 22, 2011, for DEVICE AND METHOD FOR MEASURINGSHOT FORCE EXERTED ON A MOVEABLE GAME DEVICE; and WO 2009/043558 A1published Apr. 9, 2009, for FORCE SENSOR FOR RACQUET HANDLE.

BRIEF DESCRIPTION OF THE DRAWINGS

The novel features of the invention are set forth with particularity inthe appended claims. A better understanding of the features andadvantages of the present invention will be obtained by reference to thefollowing detailed description that sets forth illustrative embodiments,in which the principles of the invention are utilized, and theaccompanying drawings.

FIG. 1 illustrates an exemplar training process showing main processesin a closed-loop system with training loop and an augmentation loop.

FIG. 2 illustrates an interaction between a stroke motion and the taskand environment elements, including the ball trajectory relative to thecourt, the impact of the ball and its bouncing before the interceptionwith the racket trajectory. The figure also illustrates the gaze of theplayer along different point of the ball trajectory, and shows a ballmachine as an apparatus that can be programmed to enable different formsof interactions.

FIG. 3 illustrates a general movement trajectory and associated primaryoutcome highlighting the movement phases, and showing the optimaltrajectory, the admissible envelope, and the feasible envelope acrossthe movement phases resulting from the various movement constraints.

FIGS. 4A-4F illustrate primary movement patterns (or movement unit)along with corresponding phase segments for different movementactivities. The drawings also highlight the quantities that can beregarded as primary outcome as a vector (effect of impacts on ball fortennis, golf and baseball, and propulsive force for running andtransversal acceleration for skiing).

FIG. 5 illustrates a forehand pronation and supination showing basicbone and muscle structures. It also illustrates possible sensors tocapture the relationship between forearm movement and musculoskeletalstructure.

FIG. 6 illustrates components of a platform that a user engagesaccording to the disclosure including the cueing system andcommunication system that are used to enable user performanceaugmentation and interactions.

FIG. 7 is a block diagram of cue processor; an overall view of thefeedback augmentation processes across four cueing mechanisms: phasetransition cues, phase profile cues, alerts and outcome validations.

FIG. 8 illustrates a temporal structure of the motion activity datadescribing the segmentation hierarchy and its typical temporal scales.

FIG. 9 is a diagram of motion skill processing components which includesmovement data processing (I), movement model (II), skill assessment anddiagnosis (III), and feedback synthesis IV).

FIG. 10 is a block diagram of cueing system showing exemplar componentsincluding the cue processor (movement phase estimator, featureextractor, and a cueing law) and cue generator (cue encoder andtransducer).

FIG. 11 illustrates a tree structure associate with the tennis strokehighlighting the movement unit classification (into categories, typesand classes of strokes) and segmentation (into phases). The phasesegments are characterized by profile and transition feature attributes.

FIG. 12 illustrates an outline of feedback levels nested according tothe human information processing organization. The figure also providesexamples of devices and system components used to process data andmediate information to the user.

FIG. 13 illustrates an overall platform architecture describing thegeneral functionalities for three example activities (tennis, skiing andskateboarding).

FIG. 14 is a block diagram overview of the movement unit profilesclassification using feature representation.

FIGS. 15A-15E illustrates the development of movement pattern structurein terms of finite-state model structure for the tennis stroke example.

FIG. 16A illustrates an ensemble of racket angular rate profileshighlighting features that can be used to identify stroke phases,including: backswing, back loop, forward swing, impact, follow through,and recovery.

FIG. 16B illustrates racket orientation trajectories (elevation andazimuth) for an ensemble of forehand strokes over a stroke cycle on aunit sphere. The figures also highlight the stroke phases. Two views arepresented to provide of the back side and front side of the stroke.

FIGS. 16C-1 to 16C-4 illustrate a 3-dimensional angular rate phaseportrait for an ensemble of forehand/backhand, topspin/slice strokes.

FIG. 16D illustrates an angular motion for several forehand strokes withdifferent topspin and slice outcomes and highlights the stroke phases.The figure also illustrates a normal envelope and an extended envelopeassociated with the back loop phase.

FIG. 17 illustrates an exemplar organization of the data processing andother functionalities across the platform's hierarchy and highlights theinformation flow across primary component levels and devices.

FIG. 18A illustrates the racket and IMU sensor reference frames with keyvariables including the measured quantities (racket 3 DOF accelerationsand angular rates) and primary estimated quantities (roll, pitch, yaw).

FIG. 18B illustrates the stroke path in terms of absolute sphericalcoordinates (azimuth and elevation) and exemplar stroke phases over atypical forehand stroke cycle. The figure also illustrates the racketbody frame (impact datum frame) and the racket absolute orientationrelative to the impact orientation.

FIGS. 19A and 19B illustrate exemplar user interfaces.

FIG. 20A illustrates a finite-state stroke model.

FIG. 20B illustrates the processes of the cueing system for top spinaugmentation profile over the stroke cycle, highlighting exemplar phasetransition and phase profile derivatives and other quantities extractedfor the computation of the feedback cue.

FIG. 21 illustrates a block diagram of the iterative training systemprocess.

FIG. 22 illustrates the control and information processing hierarchy forhuman movement behaviors, highlighting the knowledge, rule-based andsignal based levels. The figure also illustrates the sensory andperceptual components used to relevant information from the taskenvironment.

FIGS. 23A and 23B illustrate an exemplar embedded sensing and cueingdevice with the main physical components.

FIGS. 24A-24F illustrate the cueing mechanisms (outcome validation,alerts, and profile and transition cues) at different movement phasesfor tennis, skiing, golf, running, swimming, rehabilitation.

DETAILED DESCRIPTION

The disclosure is directed to a system which provides systematic,data-driven, computer-assisted movement skill training, maintenance andrehabilitation. The systems and methods use objective and quantitativemethods of skill level assessment, the use of these assessments are thenused to identify specific deficiencies, followed by the formalspecification of a training intervention. Once it is possible totranslate these components and their interactions quantitatively bydecomposing the movement into decomposed movement data, it is thenpossible to analyze the decomposed movement to identify specificallywhere change is required. This enables a user to close the training orrehabilitation loop and run it as an iterative scheme.

FIG. 1 illustrates an assessment loop 100. The assessment loop 100 isconfigurable to have five components. An extractor 110 extracts motionelements from a target motion. The extracted motion elements can bedirected from an augmentation loop 120 which collects information fromuser training or play. The augmentation loop 120 can have a feedbackloop between a movement process 122 and a cueing system 124.Additionally, the augmentation loop 120 can received information from aninstruction module 130. The instruction module 130 may receive a set oftarget skills 140 from a user or a trainer. Session data 126 can beprovided to the extractor 110. The extractor 110 output generates amotion model 150 which can then be used for skill assessment anddiagnostics 160.

A measurement process can be provided that maps aspects of behavior ormovement into one or more measurement signals. A control algorithm isconfigurable to generate a control action based on the measurement ofthe current behavior or movement and a desired behavior or movement. Anactuator that transforms the measurement signal into an input signal canbe provided that will help steer the system. The system operationalizesthe training process and creates a systematic schedule that buildsskills in following logical development, consistent with human learningprinciples. The training starts from a user's existing motor skills andproceed by shaping these skills towards the specified goal skills.

The cueing system 124 can include two components: a cue processor and acue generator. The cue processor translates movement data into cuesignals. The cue processor implements a finite state estimator and acueing law calculator. The finite-state estimator is an approximation ofthe user's movement model (which is itself represented as a finite statemachine). The cue generator translates cue signals into physicalstimuli; the system operates in real-time to provide feedback as theuser participates in an activity.

The cueing law calculator takes the state estimate and the motion dataand operates on them to calculate if a cue will be delivered and whatthe cue should communicate. The feedback synthesis model determines howthe cueing law calculator operates, whereas the finite-state estimatoris defined by the user's current movement model. The cue generator takesthe cue signal and translates it into a feedback stimuli generated by atransducer (audio, visual, haptic, symbolic, or other type). The form oftransducer is determined by the platform implementation details, usercharacteristics, equipment parameters, environment status, and/or otherconcerns.

The system receives input from a user's physical movement that takesplace during a use or play session. The measurements can capture a rangeof movement behavior that performed to complete the activity (e.g., allthe motion associated with a tennis stroke, all the motion associatedwith a golf swing, etc.), associated task conditions, as well as theelements relevant to the broader functional components such asperception of task elements.

MEMS IMUs (e.g., available from ST Microelectronics and InvenSense)typically include 6-axes acceleration and angular rates and 3-axesmagnetometers, which are often used to estimate absolute orientation inspace (Attitude and Heading Reference System or AHRS). To render usefulinformation from collected movement measurement data collected,structural characteristics are identified that can then be related toparticular motor events or actions. For computational analysis oftechnique and skills and ultimately synthesis of effective feedback fortraining instructions, it is necessary to break down movement intomovement elements.

Movement characteristics can be represented as geometrical andtopological properties, which can be related to specific aspects ofmovement organization and skill. For example, movement characteristicscan be observed in movement phase portraits such as that of the racketangular rate. Ensembles of movement data can be analyzed for patterns(e.g. using principle component analysis, phase-space analysis andnonlinear time series analysis techniques such as state-spaceembedding). In addition, machine learning techniques can be applied toanalyze the distribution of features and characteristics of themovement, as well as to aggregate and classify the data to determinepatterns which in turn can be used to determine a deeper organization ofthe overall system. Given the variety of movement types and thevariability in human performance. Typically, the system is configurableto distinguish between different movement types before proceeding todeeper analysis of any individual movement or component thereof.

One or more motion sensors, either embedded, or deployed in the user'senvironment, can be used with the system to provide measurements ofmovement dynamics encompassing one or more users, actors and theirassociated equipment (if any). As will be appreciated by those skilledin the art, given the depth of hierarchical levels of the movementsystem, the scope of motion analysis can be conducted at multiplelevels. For example, it could focus on neuro-motor aspects, movementtechnique and structure, and outcomes, all the way up to tactical andstrategic levels.

Analysis of the intrinsic movement structure of the movement techniqueand functional characteristics can be used for skill analysis. Priorsolutions have focused on the interactions of the movement withenvironment and how operators organize their behavior in relationship toenvironment and task elements.

FIG. 2 illustrates an exemplar motion in the form of an interactionbetween a user's stroke motion during a tennis game and the taskenvironment elements for a user 10 using a tennis racket 20 to impact aball 30. One or more motion tracking cameras 210, 210′ are provided. Thegaze 12 of the user 10 changes at different points depending on thetrajectory of the ball 30. The apparatus shown in the same figure can beprogrammed to enable different forms of interactions. In the tennisexample, the apparatus is a ball machine that can be programmed tosupport the development of specific stroke patterns and therefore can beprogrammed in conjunction with the cueing system.

In contrast, the systems and devices disclosed augment skills,including, for example:

-   -   (1) providing users feedback for training, including providing        signals during the performance;    -   (2) enhancing the athletic experience during performance to help        focus;    -   (3) providing protection from injury by helping users engage in        optimal techniques; and    -   (4) developing training protocols which are directed to        developing skills related to the training.

Patterning characteristics are expected in many movement activities. Intennis, for example, the same general stroke pattern can be used togenerate different amounts of top spin or pace. However, to maximizethese different outcomes, distinct patterns have to be formed to fullyuse the biological system. For example, a stroke for a top spin or slicehave characteristic features in the temporal and spatial arrangement ofmovement phases. Movement patterning is due to how changes in movementoutcomes or task conditions affect movement technique. As the desiredoutcome or task conditions change, the biomechanics and motor-controlmust organize differently to best take advantage of the system'scapabilities.

From a trajectory optimization perspective, the changes in outcome andcondition alter the system's “operating point” and result in activationof a different set of constraints. Due to the nonlinearity, this leadsto the emergence of a different motion pattern with distinct dynamiccharacteristics. Patterning corresponds to a tendency for thetrajectories in each class of behavior to stay close together in spatialand temporal terms. This closeness can be described formally usingtechniques from nonlinear time series analysis. Using these techniques,measurement data describing racket state trajectories during a tennisstroke can be aggregated and clustered to identify different strokepatterns.

Such performance data for an activity taken in its totality, for examplefrom measurements of an entire tennis match, results in a repertoire ofdistinct movement patterns. The repertoire of distinct movement patternsis the result of the optimization of movements technique, i.e.,achieving the range of outcomes and conditions required to be proficientin the particular activity. For instance, in tennis an individual willdevelop a repertoire of different strokes to optimize the necessaryoutcomes (e.g., type and amount of spin, strength, etc.) and accommodatethe range of impact conditions (ball height, speed, etc.). Thisrepertoire essentially plays the role of a vocabulary of motion patternthat an individual can call upon when engaged in a particular activity.

The patterning and repertoire, therefore, have important implicationsfor the assessment of skills. The skills of a particular tennis player,for example, can be assessed by:

-   -   (1) extracting characteristics about the entire repertoire of        strokes, e.g., how well they collectively achieve the range of        outcomes and conditions in the activity domain;    -   (2) determining how well and how consistently each class of        strokes in the repertoire achieves associated outcomes; and    -   (3) determining how well the strokes adapt to the impact        conditions.

The first analysis provides a comprehensive assessment, and the secondemphasizes the technical implementation of the motion skills.Understanding human movement control provides a deeper assessment of themovement technique and feedback that helps optimize movement technique.

Skill-based movement behaviors are usually fast, coordinated,multi-dimensional movements. Delays in human's signal transmission andprocessing limit the role of feedback. Therefore, the biologicalmovement control system has to rely extensively on “open-loop” control,meaning that trajectories are implemented from pre-programmed profilesare largely unconscious. The general motor program (GMP) explains howcomplex movements are programmed. GMP describe the generalized rulesthat generate spatial and temporal muscle patterns to produce a movementfor the collection of movement patterns in the repertoire. The GMPencompasses the mechanisms needed for adaptation to conditions.

Complex movements frequently involve a sequence of distinct movementphases.

Therefore, motor programs encompass mechanisms in order to time andsequence these elements. The movement phases are usually formed tosupport various functional characteristics, such as biomechanicalconstraints, task structure and various sensory interactions with theenvironment. Movement segments can be conceptualized as a movementdirected towards a sub-goal, each with its particular biomechanical andsensory-motor constraints. This structure allows the break down complexmovements into simpler movement elements. It can also help acquisitionof complex movement skills and support the flexibility and adaptabilityneeded to operate in dynamic and uncertain environments.

The bandwidth limitation for closed-loop feedback involving perceptualmotor control is somewhere between 0.5 and 2 Hz, depending on the task.Above that bandwidth, intermittent closed-loop can be used. Movementphases typically represent open-loop segments. Corrections can beimplemented at specific phase transition. These phase transitions arealso associated with functional features, such as when specific elementsof information are available. For example, in a tennis stroke, anadvanced player already has an idea of the intended outcome andanticipates the conditions of the oncoming ball, at the initiation ofthe stroke. At the end of the backswing phase, and before the initiationof the forward swing, the player makes adjustments based on theup-to-date information available from the oncoming ball trajectory (seeFIG. 2 which illustrates a swing trajectory).

As will be appreciated by those skilled in the art, movement skillsoften involve extensive interactions with the task and environmentelements. For example, in tennis these interactions include producingthe desired outcome in the task and dealing with the range of impactconditions. The perceptual system usually provides cues that are used toselect the type of motion pattern from the repertoire of learnedmovement patterns. Signals from the sensory or perceptual system areused to modulate specific aspects of the pattern, such as the timing ofthe stroke phases based on the tennis ball's perceived speed. Trainingmovement skills, therefore, involves acquiring a comprehensive set ofmechanisms. Movements are not simply programs to steer body segments;they encompass numerous functional features. Therefore, skillacquisition also includes learning how to extract relevant signs or cuesfrom the task environment, and developing plans for sequencingindividual movement patterns. The basic motor learning concepts areintroduced next and will expand in the later sections.

Motor skills are thought of as acts requiring integration of bothsensory information and motor responses to attain a particular goal.Goal-directed, deliberate, instrumental or intentional movements aremovements characterized by forethought with reference to theconsequences they produce. The outcome to be obtained is clear to theperformer and determines his organization of the movement pattern. Suchdeliberate movements contrast reflexes or fixed action patterns. Motorskills are categorized on a continuum defined by the dynamics of thetask and environment condition. On one end of the continuum is the openskills, which take place in temporally and spatially changingconditions; on the other end is the closed skills, which take placeunder fixed, unchanging environmental conditions.

In open skills, a new movement emerging in response to a new aspect ofthe task environment may not originate as a variation of an existingpattern. Rather, it is presented as a new movement that is formed as aunique new pattern, albeit the new pattern may be reusing components ofthe original pattern. Therefore, in open skills, the user develops arepertoire of movement patterns that match the range of environmentalconditions and task requirements. On the other hand, in closed skills,as the user learns to master the task, the movement performanceconverges over time to a fixed movement pattern that optimizes theoutcome in relationship to the task requirement.

Most complex movements are obtained by combination of motion segments orphases. The high number of degrees of freedom (DOF) in human motionresult in redundant movement solutions. For example, racket swinging canbe achieved through various combinations of joint motions such as elbow,shoulder, wrist, etc. Each DOF has its own specific displacement rangeas well as other constraints such as speed or torque. Differentexecutions of the same general movement will cause saturations atdifferent stages of the overall trajectory and will result in a sequenceof movement phases. Human users mostly learn through practice; theyessentially discover how to best exploit the rich movement space toaccomplish the desired outcome. As discussed earlier, skill acquisitionproceeds through stages that result in increasing use of the availabledegrees of freedom.

Typically, a deliberate movement is needed to produce a particularoutcome or change in the environment. Many skilled movements involve thecontrol of an end effector such as the hand, foot, or a piece ofequipment or instrument. Another important class of skilled motions arecharacterized by controlling the dynamics of interactions with anenvironment such as in skiing or surfing. These interaction behaviorsinvolve the performance of particular maneuvers to allow deliberatecontrol of motion. Examples of key maneuvers include turning orstopping. The purpose of movement skill acquisition can be defined asthe process used by an individual to learn to change or maintain theirstate or the state of objects in space.

These end-effector motions encompass a variety of different movementbehaviors including reaching motions, such as those used to grab anobject or touch something, or interception and throwing or hittingmotions. All of these motions guide the end effector along a path to aparticular location in space. Most of the reaching motions involvestationary end conditions. Interception and hitting involve more dynamicend conditions. Most skillful end-effector motions involve the precisecontrol of its state at the instant of the action (contact, interceptionor throw).

Reaching or interception motions rely heavily on visual information. Theoutput side of behavior, i.e., the control of motion, only describespart of the problem. The input side of the behavior, which encompassesthe sensory and perceptual mechanisms, is essential for a completeunderstanding. These movements are in part driven by motor program butfunctional aspects such as the adaptation of the program to externaltask elements or dynamics represents a fundamental aspect of the skills.Goal-directed movements, such as swing sports, are organized around agoal state.

In tennis, for example, the racket stroke motion is organized around theimpact, however it is achieved through a complex coordinated pattern ofarm motions that satisfy the constraints of the ball impact and the bodyand limb biomechanics. In other activities such as skiing, there is noexplicit goal. Skiers use gravitational forces and body biomechanics togenerate turning motion to steer and control their path. Thesecoordinated movements represent the primary unit of motion. There may bea great variety of movement patterns that satisfy these constraintsdepending on the configuration and conditions; however, they alltypically share characteristic features that enable the movementpatterns to be identified and undergo subsequent analysis.

Understanding the acquisition of movement skill requires data thatprovides a comprehensive description of the movement and task andenvironment elements. Most movement skills involve many degrees offreedom; tracking movement skills using early techniques meant tediouslyobserving video footage. More generally, limitations in formalsystematic skill evaluation and modeling are due to various complexitiesrelating to the fundamental nature of complex movements and other taskenvironment characteristics.

Skilled human movements, such as the tennis stroke, involve thesequencing of complex coordinated motions that are executed based oninternal states and external cues. Their successful performance involvesmanaging a range of contributions, including the effects of the tool orequipment (e.g. the tennis racket), the movement biomechanics, theinteractions associated with the activity (e.g. tennis ball impact), andthe interactions with the environment (e.g. aerodynamics or othermedium).

The human motor system evolved to manage these interactions andefficiently handle the variety of uncertainties and disturbancesprevailing in the task and environment. However, while the humanmovement system has tremendous potential, systematic and dedicatedtraining is required for high levels of motor facility. This is similarin any domain of activity, such as athletics, music, or vocational.These factors can be divided into extrinsic and intrinsic factors.Extrinsic factors include the interactions with the environment such asthe foot strike or impact of the ball on the racket. Intrinsic factorsinclude the biomechanics, human motor control, and effects arising fromthe manipulated equipment's dynamics.

Most skilled behaviors are so-called deliberate behaviors that aredirected at achieving specific outcomes. Therefore, learning skilledbehavior in sports or vocational activities involves learning to masterthese interactions so as to achieve the desired outcomes or goals. Thecoupling of the human movement system and the task environment have tobe considered as a coupled system. If the extrinsic and intrinsicinteractions were considered separately, the complexity would beintractable. The key to a tractable solution, therefore, are strategiesthat structure and organize movement behavior to simultaneously dealwith the overall system.

The brain evolved specific organization and functionalities toefficiently work with these complexities. The brain and sensory-motormechanisms that optimally deal with the coupling of the two domains, andachieve the sufficient level of adaptation, have been determined byevolutionary process. As a result, the specific structure andorganization of the brain, including the nervous system and largerbiomechanical systems, support natural solutions that enable efficientand adaptive behaviors. Therefore, a portion of the movement system isgenetically determined. However, motor skills, especially in deliberatemovements, are learned based on repeated interactions within the taskand environment and are best acquired early in life when the brain isstill developing. Finally, learning movement skills involves changes inthe cortex as a result of neuroplasticity. These changes, however,follow a specific process that is dictated by the organization of thevarious cortical structures (cerebellum, parietal cortex, pre-motor andmotor cortices, and the prefrontal cortex).

Three key forms of units of behavior have been described for complexmovement behavior. At the top level, motion primitives are related tothe concept of “motor equivalence” which has been identified as one ofthe fundamental characteristics of motor behavior. The idea is that thesame movement behavior can be repeated in various contexts and withoutchanging the overall form of the motion. Therefore, segmentation ofhuman movement behavior into motion primitives has been most successfulby identifying invariants characteristics. Since complex movements areobtained from a sequence of movement phases, the next level of primitiverepresents the patterns that can be combined sequentially.

The elements of the movement system are derived from structural featuresextracted from measurements using pattern analysis. Muscle synergies areobtained from factorization methods (e.g., principle component analysisor non-negative matrix factorization). The general idea is that manymovements can be described as variations of a general model and once thegeneral category of movement is specified some of the key mechanismsneeded to achieve robust movement performance are those that allowadapting to changes in conditions or transferring to new task oractivity.

In contrast to periodic and reflexive movements, which can be generatedfrom low-level brain functions, skilled movements usually involve thedeliberate formation of specific goals or outcomes. These movements maybe completely self-initiated, e.g. picking up the phone to call someone;they may represent a stage in the context of a larger task, e.g. openingthe dish cabinet when preparing food or returning a tennis serve. Asseen in these examples, movements are rarely an isolated behavior butare part of a larger set of interactions with the world and thereforeare typically part of a chain of behaviors.

Learning deliberate skilled movements involves learning the perceptualcues and motor actions that successfully accomplish the intended goal oroutcome (e.g. reaching to grab an object or imparting momentum to aball). Learning involves iterating on these solutions as the task, or asimilar task, is repeated.

As will be appreciated by those skilled in the art, teaching relies ontwo primary modalities: demonstration and practice. Demonstration intheory should focus on instructions to help the student understand whatthey need to know about the behavior or movement. Practice refers to theprocess of performance repetition. Studies have shown thatdemonstrations too often focus on the task outcome rather than on ananalysis of the coordination of movement. Movement skill acquisitioncould therefore be accelerated by providing specific signals deliveredduring performance. Two signals in particular would be beneficial.First, signals that highlight the structural elements used in thecomposition of movement. Second, signals that indicate whichcharacteristics of these elements play a role in movement outcomes.However, these signals have to be adapted to the skill level of theindividual and to his or her specific movement technique.

Users form an abstract understanding of movement capabilities in termsof goals and outcomes. Users, for the most part, learn in which contextsto use and initiate movements. Therefore, at the highest level, peoplecan assess how well they do from knowledge of their repertoire ofmovements. The technical details of movement skills are largelyunconscious. This is in part because movement execution is too fast forhumans to directly control their technique. Therefore, most learningsfollow a trial and error process. Movements that achieve goals areessentially reinforced.

It is difficult or impossible to directly assess movement technique.Users typically only determine technique indirectly through its outcome.Therefore, it is hard to explicitly instruct the technical aspects ofthe motion skill system. Trainers and coaches increasingly usestrategies to help users form sense memory associated with a correctmovement technique. A feedback signal that validates correct movementfeatures can, through association, be used to reinforce memory of whatsuch a movement should feel like. This form of feedback should henceaccelerate the development and consolidation of a particular skill.

There are three primary forms of feedback which operate at specificlevels of the control hierarchy: Real-time feedback, taking place duringperformance; feedback immediately following an action, such as based oninformation from the movement outcome; and feedback at the end of atraining set or session. Inherent feedback associated with the feel,look, sounds, etc., of movement performance, as well as the movementoutcome and interactions with the task and environment, provide atremendous amount of information that can be used to assess performanceand help train. Individuals, however, have to learn to recognize andevaluate those sources of information. Augmented feedback is informationthat is supplementary to inherent information about the task ormovement. The two major categories of augmented feedback are: knowledgeof result (KR) and knowledge of performance (KP). KR representspost-performance information about the outcome or goal achieved. It issometimes called reinforcement. Note, however, that not all movementshave an outcome that is separable from the movement performance. KPrepresents information about the movement technique and patterning. Thisinformation is useful for the acquisition of complex movement skills,such as those requiring high-dimensional spatial and temporalcoordination. Previously, it was difficult to measure and trackperformance in many activities. The advent of MEMS movement sensors hascreated a wide range of possibilities for using information aboutmovement kinematics and dynamics (kinetics) from measurements.

Several levels of feedback are involved in support of skilled movements.Feedback acts at different levels across the hierarchical organizationof the nervous system. The cortical and subcortical functions involvedin the formation and implementation of movement patterns and thedifferent feedback structures used to correct and modulate the movement.At the lowest level the spinal and subcortical system used for thephysical implementation. This system receives commands from the corticaland subcortical structures.

Feedback encompasses the information sensed at the level of the muscles,tendons and joints and modulates at the level of spinal circuits.Between the spinal and subcortical is the system that controls posture.The feedback encompasses information from the vestibular andproprioception combines spinal circuits with cerebellum. At the centerof the system is that formation of complex movement patterns, especiallythe chunking and sequencing of movement phases. Feedback mechanisms useinformation from cues extracted by visual, auditory, haptic sources.

The task of this system is to fine tune and synchronize behavior withexternal task and environment elements, such as adapting timing ofmovement phases, or modulating phase profiles. The phases are typicallypart of a sequence generated by the cortical circuits. The higheststructure is the cortical system used for perception, planning andexecution. This system combines the various sources of sensory andperceptual information to build representation that can be used togenerate plans and monitor the performance and outcomes of the behavior.This system can handle more abstract forms of information such as verbalor written.

Skill-based behavior is driven by signals that carry the proprioceptiveinformation as well as unconscious exteroceptive information.Skill-based behavior involves the automated sensory-motor system.Rule-based behavior is driven by signs or cues, which typically describea state of the environment, and determines which stored pattern toactivate. Knowledge-based behavior is driven by symbols, which describeabstract states related to a conceptual representation and serve asbasis for reasoning or planning (see FIGS. 23A and 23B).

The human information processing model helps understand the type offeedback information and associated components of movement behavior.Table 1 summarizes the type of signals, cues/signs and symbols in tennisas an example. At the highest level, the knowledge-based behaviorcorresponds to the type of stroke and body positioning, etc. to usegiven the information about the overall situational awareness, such asadversary behaviors gained from exteroceptive information. At theintermediate level, cues trigger behavior. At the lowest level, signalsare used to modulate muscle responses.

TABLE 1 Example of signals, cues/signs and symbols in tennis. Symbol:The type/class of stroke as well as the desired ball placement. Cue/signWhen to initiate a stroke phase and the modulation of the stroke patternbased on the predicted ball impact and bounce trajectory, etc. SignalCoordination of the muscles during the stroke to conform to learnedpattern based on proprioceptive feedback.

At the highest level, the rule-based behavior involves determining whichpattern to activate based on the signs or cues typically obtained fromthe exteroceptive information. At the intermediate level, cues are usedfor time movement execution. For example, the particular state of theball extracted visually, such as the impact, may be used to signal theinstant to initiate the backswing or the forward stroke, and modulatethe strength of the initial acceleration. Finally, at the lowest level,the skill-based behavior corresponds to movement patterns.

Signals are primarily the proprioceptive information. The delays andtime constants of the sensory-motor system are too large to providecontinuous feedback corrections for fast-paced skilled movements. Theneuro-muscular time constant (time from the signal to go from the motorcortex to the muscle response) is of the order of 20-30 msec; on theother hand, the response time from a visual or auditory stimuli to aphysical response is of the order of about 200 msec. Therefore, skilledmovements rely on open-loop execution. Feedback is structured, forexample, for intermittent actions based on specific cues and controllingthe timing of phases.

The largely open-loop execution implies that segments have to be learnedin order to be reproduced accurately, and mechanisms to predict theoutcome of the movement are necessary to enable modulation of themovement at the instant of execution. This general model can be appliedto augmentations used to assist or train human movement behavior. Inprinciple, augmentations can be designed across all three levels. Motionskills, however, assuming the outcome is known, primarily involve theskill-based and rule-based behaviors.

The forms of KP feedback that are most useful are those that contributeto the understanding of the task or movement. This explains whyproviding a reference trajectory, to model after, is not necessarilyuseful. In that sense, KR has the advantage that it provides objectiveinformation about the implicit correctness of a movement.

Since human attention capacity is limited, it is important to select theaugmentations that also account for these limitations and possiblyorganize the augmentation in ways that allows the brain to takeadvantage of the mechanisms used to operate efficiently with information(e.g. chunking).

Creating KP feedback contributes to understanding the task or movement.This can be achieved by using movement kinematic and dynamicmeasurements that produce knowledge of performance that is connected tothe movement outcomes, as well as organized in terms of timing and form,etc. in ways that are consistent with the movement's functionaldimensions, including biomechanics, motor control and sensing orperception mechanisms.

The core component of motion analysis and cueing platform technology isthe decomposition of movement into elemental movement units that aregrounded in biomechanics and motor control principles including musclesynergies. This makes it possible to generate feedback that connectstask outcome to requirements on these elements of performance. At thesame time, using feedback that is structured based on movementorganization will ostensibly help better overcome attention limitationsince the movement units are part of a coherent movement language. Witha technology that will reinforce and teach this language, it can helpacquire a form of movement intelligence that is difficult to developusing an ad hoc notation system.

By working within natural functional elements and features, it ispossible to factor out effects due to individual differences. Focusingon the structural characteristics of movements derived from performancedata, and subsequently identifying features within the functionalelements that contribute to the outcome, makes it possible to designfeedback augmentation that targets individual movement characteristicsbut generalizes over the range of skill and styles as well asdifferences that can arise due to injuries and other factors.

Complex movement behavior can be sequenced into movement elements. Mosttasks are composed of a series of stages and complex movements aredelineated in distinct phases. Therefore, to analyze movement skills, itis necessary to understand how movement elements are composed. First,how complex movements are achieved by sequencing movement segments, andsecond how the individual segments are achieved through coordination ofdifferent body components.

The first is a temporal decomposition into subtasks or subgoals thatdepends on the elements of the task and environment. The second is adecomposition into spatial components that can be combined in parallelto achieve a library of coordinated movement patterns. The movementpatterns themselves often involve a sequence of phases.

Complex human movements are high-dimensional, i.e. their descriptionrequires large numbers of state variables (position, velocities,angles). The representational complexity is in part due to 3-dimensional(3D) space which involves 6 degrees of freedom for the linear andangular motions. This number gets multiplied when multiple body segmentsare involved and becomes exponentially complex once the ligaments andmuscles need to be accounted for.

In addition, there are the effects of dynamics, which dictate how thesestate variables evolve over time and interact through the action offorces (both internal effects such as inertial coupling and the externalactions such as the muscles or aerodynamics, etc.). For this reason,even tracking a single segment or object in 3D space such as tennisracket or forearm, requires a dozen state variables. Their temporalevolution is described through coupled differential equations. Thesedifferential constraints and other constraints on joint configuration,etc. result in geometrical properties which can be exploited foranalysis.

From a control standpoint, the formulation of movement programmingtypically follows the specification of equations of motion, an initialstate and a goal state. Such problems can be solved as a dynamicprogram, or a two-point boundary value problem. The trajectory isobtained by solving for a trajectory that minimizes a pre-specified costfunctional (e.g. trajectory duration or energy). This formulation leadsto equations which provide conditions for an optimal trajectory. Thus,for a given initial state and goal state (e.g. that specifies theoutcome), there typically exists a unique optimal movement trajectory.

The control and trajectory optimization framework provides useful toolsfor the conceptualization and analysis of movement. For example, it ispossible to define cost functions that can help characterize humantrajectories, such as measuring energy or more general physicalperformance. Furthermore, the calculus of variation used in trajectoryoptimization make it possible to investigate relationships betweenvariations in trajectory and outcomes of the trajectory.

The three primary levels of movement organization are:

-   -   (i) the movement profiles and their associated outcomes. This        level corresponds, for example, to task level description and        represents overall movement element or unit such as a tennis        stroke in tennis.    -   (ii) the movement profiles are usually composed of series of        multiple phases. This level corresponds to the biomechanical        implementation level, i.e., how the limb segments and joints are        coordinated to achieve a complex movement.    -   (iii) the movement phase profiles can then be decomposed into        muscle synergies.

This level corresponds to the neuromuscular implementation, i.e., howthe profiles are achieved by superposition of muscle units. The musclesynergies represent muscle activation patterns.

The first, corresponds to what could be considered the semanticcharacteristic, i.e. the general movement technique that can be used toachieve the intended outcome, and is related to strategies developed bythe brain to partition the workspace and achieve a range of outcomesrelevant to a task.

The second, the phase segmentation, corresponds to the internalstructure of the movement, and is related to the strategy used by thenervous system to achieve the particular outcome given the availableneuro-muscular system.

The third, the muscle synergy, describes how the various muscles areactivated to achieve the movement profile at the phase level. Thesynergies typically provide spatial and temporal components that can becombined to achieve a variety of movement. Therefore, it is expectedthat same set of synergies can be reused by other movements. Yet, forexample, in tennis the arm segments configuration can be very differentat different stroke phases, therefore it is likely that different setsof synergies are used in each phase.

Movement measurements, such as from wearable motion sensors or opticalmotion capture systems, represent nonlinear time series. The analysis oftime series relies on understanding the structural characteristics ofthe underlying dynamics. The characteristics associated with thearchitecture of the movement such as the movement phases in a tennisstroke or golf swing. Insights can be gained using computationalvisualization tools such as phase space; however, the states may havetoo many dimensions to be practical. Therefore, the data should bereduced. The behavioral data captured from the available measurementsresults in a high-dimensional state space. The dynamics driving thebehavior, on the other hand, may be lower-dimensional. Dimensionalityreduction can be used to discover the dimensions underlying behavior. Itbelongs to the class of unsupervised learning technique.

For nonlinear time series, the goal is to transform the originalmovement data, which are described in terms of the high-dimensional timeseries x_(t) into a lower dimensional description that preserves thegeometric characteristics of the movement dynamics. This can be done,for example, using Taken's embedding theory. Examples of recentapplications of dimensionality reduction include gait analysis.

In some applications, movement phase segmentation can be based onpre-existing or empirical understanding of the movement phases. Theaccuracy of the movement phase decomposition, and the validity of theassociated finite-state model, can vary significantly from oneindividual to another or even on the class of motion considered. Sincethe movement dynamics and control processes are nonlinear, a broad rangeof movement patterns can exist for the same intended movement andoutcome. The specific movement structure will depend on the individual'sskills, technique, the particular body characteristics, biomechanics,and other factors including injury, disease, etc. Therefore, the phasedecomposition requires a method to determine the optimal movementsegmentation and its finite-state description.

FIGS. 15A-15E illustrate the development of movement pattern structurein terms of finite-state model structure for a tennis stroke, as anexample.

Although movements are usually high-dimensional behaviors, trainedmovements typically have specific patterns. Patterns have the usefulproperty that even though the behavior relies on many degrees of freedom(DOF), they can be described by a few, dominant DOFs. The patterns forma lower-dimensional system as a result of the coordination provided bythe neuro-motor processes, and other perceptual and control mechanisms.The lower dimensions however can hide a complex geometry and topology.

The movement architecture can be analyzed by focusing on the lowdimensional manifold that are associated with the particular movements'dynamics pattern. Using a nonlinear dynamic systems formulation givesaccess to analysis and modeling tools that under certain conditions canreconstruct the pattern dynamics from measurements of the behavior. Thereconstructed dynamics can then be analyzed to determine the structureand geometry which can then be used to determine useful abstractions ormodels.

Using mathematical tools used for the analysis of nonlinear dynamicsystems, the movement patterns can be described by a nonlinear mapping Fassociated with discrete-time nonlinear dynamics:

x _(t+1) =F _(t)(x _(t) ,t,ϵ _(t))  EQ. 1

where F_(t) is a map, x_(t)∈

^(n) is the state vector at discrete time t∈

, and ϵ_(t) is a time-dependent noise. In the forthcoming discussion itis assumed that the dynamics are autonomous and use a constant mapF_(t)=F.

The nonlinear model of movement patterns therefore can be described by amap F. The map captures the combined effects of the biomechanics,sensory, motor-control processes. This model assumes that the movementsthat are learned result in deterministic dynamics. A continuous timerepresentation could also be used, in this case the dynamics are givenas an ordinary differential equation (ODE) {dot over (x)}=f(x(t), ϵ(t)),which describes a vector field and is typically called the flow. The setof initial conditions which result in the same asymptotic behavior arereferred to as the basin of attraction. Such a nonlinear dynamic modelscan describe a broad range of phenomena. The model could be decomposedinto subcomponents giving access to the various contributing systems andprocesses. For example, it may be possible to explicitly model how theusers adjust their movement pattern to changes in conditions such asadjustment of a forehand topspin stroke for changes in ball height atimpact. However, at this point in time the behavior is regarded as aclosed-loop behavior which abstracts the various internal mechanisms.

The language of nonlinear dynamic systems make it possible to describethe collection of movement patterns that composes a user's repertoire ina particular activity (tennis, skiing, surgery, etc.) by a collection ofdistinct dynamics or maps {F_(α), F_(β), . . . , F_(γ)}. In manynonlinear time series, the movement system state variable x is generallynot directly observable. Instead, measurements y are acquired forexample through motion sensors. The observations, or measurements can bedefined as: y_(t)=h(x_(t), η_(t)), where h is the output map and η_(t)is measurement noise.

A property of movement at the highest level is referred to as “motorequivalence”. The fact that the brain generates movements that areequivalent in terms of their accomplished outcomes underscores the ideathat at the highest level the brain encodes outcomes and theirrelationship with task goals. The planning and monitoring functionsassociated with goals are part of the brain's executive system. Forexample, in tennis, the player selects a stroke type based on thedesired outcome and the conditions (ball state including expected impactheight, velocity and spin of ball). Even within the continuum ofconditions and outcomes, it is possible to recognize distinct classes ofstrokes. The invariant characteristics in movement features enablesdelineation between movement classes, e.g., movements within oneparticular class can be related through some smooth transformation suchas rigid-body translation and rotation, i.e., they are invariant underthis class of transformation. The overall movement class can besubdivided into subclasses. For example, a hierarchical decompositionwould group movements based on relative similarity.

In tennis, the overall stroke class can be subdivided into dozens ofsubclasses based on movement where the levels represent different typesof features. For this example, a top hierarchical level is called thecategory level. It differentiates between groundstroke, volleys, serves,etc. The distinction between stroke categories is made primarily basedon the height of the impact point. Further, subcategories can be createdbased on the side of the impact, i.e., forehand or backhand. Evenfurther subclasses can be delineated based on the outcome (topspin,flat, slice), and strength. Beyond these common classes, finerdistinctions can then be added based on additional aspects of stroketechnique, such as open or closed stance.

Most of the stroke characteristics can be determined entirely from theracket trajectory and therefore do not require additional measurementssuch as the position of the player on the court. Each movement patternclass in a repertoire has a different shape and may occupy a space ofdifferent dimension. The shape and dimension is a result of thedynamics, which is given by the transition map F. The repertoire is thecollection of these shapes. From a mathematical standpoint, thegeometrical characteristics of the movement patterns can be describedvia embedding theory. The idea is to determine the subspace of DOF thatfully describes the movement. The dimensionality of the system and thegeometry of the manifold that contains the trajectory describe themovement class structure.

As is often the case in nonlinear system dynamics, the state transitionmap F (the dynamics), the output map h and the dimensionality of thestate vector n are not known. Techniques of nonlinear time seriesanalysis can (assuming deterministic dynamics F and smooth output map h)estimate the dynamics associated with a movement pattern from timeseries obtained from measurements of the behavior. Repertoirecharacteristics can provide a variety of information about skill.Movements are typically analyzed in specific classes withoutconsiderations about the overall repertoire structure. The movementrepertoire for a particular activity domain describes how a userorganize the outcomes and technique that task domain. The simplest wayto classify movements into repertoire is to extract features from thetime series and apply clustering techniques to determine classes.

Movement classifications have been used in other applications unrelatedto skill modeling such as activity detection or gesture recognition.Gesture recognition is a growing aspect of natural human-machineinterfaces. The general goal in the latter application is to determinemotion primitives that provide a low-dimensional description of thevarious movements that can occur in that domain. The primitives can thenbe used to classify the movements. The library can then be used by otheragents to identify the intent of a human or robotic agent and forexample allow collaboration between agents. The emphasis of gestureclassification is the identification of semantic characteristics. In thepresent application the goal is to classification based oncharacteristics that are related to movement technique and outcomes.Typically, the higher categories of the stroke classification can beconsidered in a semantic sense (e.g. groundstroke vs. volley or backhandvs. forehand) the lower level classes are related to technique.

A particular ensemble or repertoire of patterns in a domain of activityarise through the effects of biomechanical, neuro-muscular constraints,as well as task related constraints. In the most general sense, thepatterns describe how and individual's movement techniques are used toachieve an outcome. An important aspect of the movement characteristicsis how they are broken down into phases. Overall movement patterncharacteristics, therefore, are the result of the phase structure.

The serial order in behavior and the movement phase structure aredistinct. The serial order is associated with the activity level, forexample, characteristics related to the activity constraints such asprocess stages, rules, etc. The movement phase is associated withmovement technique and is related to characteristics of the movementsystem. For example, in tennis, the stages include serving, then movingto the anticipated, making adjustments in the displacements as the ballreturns, setting up for the stroke and engaging the ball using thestroke type required for the desired outcome. The overall sequence ofstages corresponds to the serial-order of behavior. Each stage can beparsed and the movement associated with it can be analyzed. The phasestructure of a movement defines the topological characteristics of themanifold and the dynamics that drive the phases define its geometricalcharacteristics.

Many complex movements are achieved by combining several movementphases, leading to further temporal structuring of the movement.Examples include the phases in locomotion gait or the phases in a tennisstroke. Phase structuring of patterns typically arise from the intrinsicmovement constraints (biomechanics), some aspects of task constraints aswell as functional factors related to motor-control and decisionmechanisms as discussed elsewhere. For example, in gait, distinct phasesare associated with the basic leg biomechanics and mechanics of groundinteractions.

In tennis, the general goal of the user is to return an oncoming ball.This is accomplished by imparting forward momentum and spin to the ballwhen the user hits the ball. The user controls the ball by modulatingthe amount of linear and angular momentum imparted to the ball. Foraccomplished players, the overall tennis stroke motion encompasses thekinetic chain formed by the legs, hips, shoulder and elbow and wrist.These segments are coordinated to form a continuous movement startingfrom the backswing all the way to the follow through and recovery. Atcloser inspection distinct phases can be recognized. The exact phasecharacteristics depend heavily on skill level.

Beginning players primarily swing the racket from the shoulder withoutvery precise coordination with the rest of the body segments. Advancedplayers exploit the entire body kinematics to maximize the outcome.Ultimately, the phase characteristics reflect the combination of thebody segments' biomechanics and neuro-motor strategies, including themuscle synergies, which achieve the highest outcome reliability withbest use of the physical capabilities. Different phases are associatedwith different biomechanical functions. For example, in walking,synergies that are activated at specific phases of the gait cycle (e.g.forward propulsion, swing initiation, deceleration, etc.) have beenidentified.

The role of constraints in creating distinct movement phases can beexplained using concepts from constraint optimal control. In optimalcontrol trajectory segments are related to the concept of singular arcs,which correspond to segments where different set of constraints areactivated by the trajectory. In general, these systems are bestcontrolled using switched control laws. The control law is determinedbased on a partitioning of the system's state. As the system is drivenby the control action, and travels through the different partitions ofthe state-space the control strategy switches to best account for thelocal characteristics of the dynamics.

Following the nonlinear dynamic systems description, trajectory phasingcan be described mathematically as a sequence of deterministic dynamicsF₁, F₂, . . . , F_(N). The overall trajectory is obtained by a series ofinitial values and asymptotic behaviors, where the next set of initialvalues corresponds to the terminal values of the previous phases (FIG. 3). The dynamics associated with each phase result from different jointand limb segment configuration and force fields. Each dynamic F_(i), cantherefore be assigned a state-space region specified by an initial stateset and a goal or subgoal set.

For example once the dynamics are initiated from the initial set, theinitial dynamics F₁ will take the state to its subgoal set

₁, and from there, assuming the state satisfies the next initial stateconditions for the next dynamics F₂ the system will switch to the nextphase, etc. The state values associated with the switching betweendynamics, e.g.

_(1i)=

_(0j) define the phase transitions. The dynamics associated witch eachphase result from different joint and limb segment configurations andforce patterns. The force patterns result from the spatio-temporalmuscle activation patterns, i.e. muscle synergies.

The synergies describe the coordination between the different musclegroups and limbs segments that are used to implement movements. Thesynergies are a type of motor primitive which is typically reserved forthe neuro-muscular coordination. In examples discussed earlier, variousmovement profiles observed in an activity can be obtained through thecombination of such primitives. Decomposition into synergies thereforecan help gain insight into the set of biomechanical and neurologicalcomponents that participate in movement skill. In turn this informationcan be used to gain understanding about the biological components, andcould be useful for physical performance, injury prevention.

Popular techniques are non-negative matrix factorization. Synergies havebeen characterized with a variety of measurements, including movementprofiles of the end points, joints and/or body segments, as well asmuscle and neurological activity such as provided by surfaceelectromyography (EMG). The type of measurements obviously determinesthe accuracy of the results. For example, simple end point or bodysegment measurements may not provide synergies that correlate stronglywith the neuro-muscular activity. Synergy analysis has not yet beenintegrated in clinical settings where it could be used for assessmentand rehabilitation. Since synergies have been identified at differentlevels of the neuro-motor hierarchy (motor cortex for grasping, brainstem for posture and spinal cord for locomotion), the muscle synergyanalysis can provide a more precise picture of neuro-motor deficits.

Some movements have an explicit outcome or goal. This goal may be themovement's end state; i.e.,

_(goal)=

_(N), or it could be the state at an intermediate phase such as asubgoal. The latter is the case for the tennis stroke. While the ballimpact is the primary goal or outcome of the stroke, this phase is notthe actual end of the movement. The movement phase following the impact,the follow through, is an important part of the overall movementpattern. Most complex movements involve many body segments or degrees offreedom. Therefore, the state trajectory is a multidimensional statevector and it is necessary to add distinctions between the differentstate trajectories that participate in the action.

Focal and corollary movements are distinguishable; the focal movementis, for example, the finger movement that hits the key; the corollarymovement is, for example, the motion of all other fingers that are partof the overall kinematic pattern essential to the task of hitting thekey. Although not every movement behavior has an explicit goal oroutcome. For example, most of the movements used in skiing have aspurpose to control the skiers' speed and direction. From a dynamicsystem standpoint this goal involves generating a centripetalacceleration though the interaction of the skies with the terrain.Depending on the skier's state and terrain conditions, different motionpattern of the legs and hips, etc. are used to achieve the best outcome(will be discussed elsewhere). For a deterministic autonomous dynamicalsystem, trajectories from a given initial state are unique, and,assuming the system has a stable asymptotic behavior, re-injecting thesame initial state will lead to the same trajectory pattern.

It is possible to define an optimal trajectory that takes the systemthrough the phase sequence achieving the goal condition (outcome) whileminimizing a performance objective such as jerk or energy. Given thebiomechanical constraints, muscle synergies, etc. the optimal trajectoryis associated with a specific phase sequence. The conditions at thephase transitions, i.e., the set of initial states, and subgoal states,

_(1i)=

_(0j) as well as the dynamics F_(i) describing the transitions,represent characteristic features of the optimal trajectory.

Absolute optimal trajectory, which is the global optimal solutions for agiven outcome, while the local optimal trajectory is for a given phasestructure. The latter, for example represents, situations where due tolack of flexibility, skills or injury, only a limited set ofconfigurations can be achieved as well as limited muscle force fields.

In optimal control theory, perturbation of the initial value leads toneighboring optimal trajectories. This is guaranteed if the initialvalue is within the so called basin of attraction of the system. Asimilar idea can be used for perturbations in the dynamics F. Suchperturbed dynamics lead to slightly different asymptotic behaviors,however, for small enough perturbations the trajectories stay closeenough to the nominal trajectory that these perturbed trajectoriesbelong to the same movement pattern. The range of perturbations in theinitial values and dynamics for which the trajectories remain in thebasin of attraction defines the admissible envelope. Perturbations inthe dynamics and disturbances are captured by the time dependent noiseterm ϵ_(t)

FIG. 3 illustrates of a trajectory envelope 300 for a hypotheticalmovement pattern. The movement in this example includes an intermediategoal set. The trajectory envelope 300 delineates a region of thestate-space over time and highlights feasible envelope 310 and theenvelope of admissible trajectories 320 as well as the envelope of theoptimal trajectory 330 (x*oi) and the optimal trajectory 350 (x*(t)).

The structure of the movement both in terms of patterning and the phasesegmentation are given by its spatio-temporal characteristics. Movementcharacteristics are defined by the geometry and dimension of themanifold containing the trajectory. Several phases are shown including:movement initiation, phase 1, phase 2, an intermediate goal phase, afollow-on phase and recovery phase. These movement patterncharacteristics are usually determined from the topology of the movementpattern manifold obtained from analyzing the nonlinear time series.

A user may choose “admissible movements” that belong to the samemovement pattern and still reach the goal conditions or outcome. Thiscould happen due to changes in movement goal conditions (impact heightand velocity), or imperfect initiation of the movement. The suboptimaltrajectories can still reach the desired end state or outcome, however,they will typically require more physical effort, may cause stress insome of the muscles or joints, or other undesirable effects. Thephysical performance can be described through models of themusculoskeletal system and cost functions such as for energyconsumption.

Movements belonging to the same pattern can therefore be related throughperturbations relative to a nominal trajectory. Moreover, the trajectoryperturbations also result in perturbations in the primary outcome andany other secondary outcome characteristic such as the different phaseoutcomes. Using this data, it is therefore possible, for example throughregression analysis or sensitivity analysis, to determine relationshipsbetween the trajectory perturbations (what correspond to the movementtechnique) and perturbations in outcomes. This information provides aquantitative basis to generate skill characteristics, such as whataspects of the technique contributes favorably to the outcomes andvice-versa what aspects are detrimental to good outcomes. This knowledgein turn can be used for training and eventually help synthesize feedbacklaws for real time cueing.

By modeling movement patterns as a sequence of phase segments withdistinct dynamics F, the pattern dynamics can be abstracted as afinite-state model. In the present case, the finite states are theindividual phase dynamics F which take the system from initial valuex_(i0) to the next subgoal state x_(i1). More generally, the initial andsubgoal states are represented by sets to account for the variations anddisturbances that are typically expected in human behavior. With thismodel, the overall motion behavior is then given by some finite-stateautomata which gets triggered from the initial state and initialmovement phase. The motion behavior combines both continuous dynamicsand discrete variables that capture phase transitions and mode switchingwhich may be associated with discrete decision variables. Hybrid modelscan be used in many modern engineering applications including roboticssuch as for autonomous systems, as well as, human-machine systems. Oncethe structure of the motion is characterized it can be described byfinite-state models.

Statistical models, in contrast to deterministic models, where thecurrent state uniquely determines the evolution of the system (i.e.,within the disturbance or model uncertainties), describe the evolutionof the probability density of future states. Statistical models such asDynamic Bayesian Networks have become increasingly popular indata-driven approaches. Popular applications in the movement domain areidentification of human activities. These approaches typically requirelearning the phase of activities based on statistical pattern analysis;and subsequently, using this knowledge to discretize the state spaceinto discrete states; and finally, determining the state-transitionprobabilities. A common model is the Hidden Markov Model (HMM). Most ofthe notational system focus on the discrete game structure and can beused to analyze game plans but currently do not reach down to the actualmovement skill level.

Real-time movement phase estimation can be implemented by someonetrained in the art. For example, a multi-layer HMM application tomovement could be based on similar models to those used for real timespeech recognition. Decoding sound recording for speech recognitiontypically proceeds in multiple levels. Most of those are associated withthe levels of organization of the speech production system. The units ofdecomposition of speech is based on phones which combine to form thephonemes. The phonemes are the basic building blocks used to form words.The phones are related to features of the vocal movements. Followingthis model for movement correspond to have, at the top level, a movementphase model which describes the probability distribution over possiblesequences of movement phases. At the midlevel, a phase model thatdescribes the composition of the movement phases in terms of movementcomponents (c.f. synergies). And finally, at the bottom level, themovement model that describes the movement components based on featuresin the available measurements (IMU unit or other sensors).

Because of individual differences in anatomy, style and skill level,movement targeting the same general outcome can be quite different. Thedifferences can manifest in the movement phase structure. For example,for a beginning tennis player, a forward stroke will be a rudimentarymovement consisting primarily of a forward swinging motion implementedfrom the shoulder joint. Over the course of skill acquisition anddevelopment, the brain will learn to better take advantage of thephysical potential, range of coordination of the body segments, andother movement system components.

Complex movements involve coordination of multiple body segments; somesegments are tightly coordinated while others can be independent such asfingers. The evolution in movement technique as movement skill developsis associated with a changes in multiple aspects of the movement system,including, a development of the physical performance (e.g. musclestrength). The most critical for skill analysis is the acquisition ofcoordination, which involve changes in the central nervous system. Asmovement skill develops movement patterns undergo profound changes inarchitecture. Therefore, the analysis of skill has to capture theunderlying structural changes.

A characteristic of learning complex movements is learning to use andcoordinate the large number of degrees of freedom afforded by the body.The structural changes in the movement can be captured by modifying andextending finite-state model. The movement phase structure results amongother factor from the effect of various constraints and functionalrequirements associated with the task. One aspect of the evolution ofmovement phase structure can be related to the concept ofco-articulation that was introduced in speech production. A task thatrequires a sequence of movement elements, training can lead to theformation of new movement elements (see FIGS. 15A-15E). These are formedthrough the interaction of neighboring elements. The optimizations ofthe links between action phases have been studied in simple tasks. Theco-articulation, however, mostly focuses on the process linking betweenelements.

The range of motion sensors, available either as embedded or deployed inthe environment, provides measurements of broad aspects of the movementdynamics of users, actors and their equipment. Given the depth ofhierarchical levels of the movement system, the scope of motion analysiscan be conducted at multiple levels. For example, it could focus onneuro-motor aspects, movement technique and structure, outcomes, all theway up to tactical and strategic levels.

Detailed analysis of movement skill, in particular for open motorskills, quickly become complex. In tennis, for example, the strokemotion is part of a much larger system of coordination and interactionsthat include the ball trajectory, the footwork, going all the way tocourt motion, the game tactics, etc. Analysis of the stroke motionusually encompasses the racket trajectory (i.e. end effector orequipment), even though that trajectory is the result of a kinematicchain which involves the upper body and the driving motion that startsfrom the feet, legs, and hips. Therefore, many elements and bodysegments should be tracked to provide a complete description of movementperformance.

Viewed through direct observation, there is typically significantvariability in human performance on repeated trials, making it difficultto apply quantitative models that describe an individual's technique andskill comprehensibly and with specificity. In addition, the movementtechnique of users is highly individualized due to specific body types,physical fitness and skill level. Therefore, it is essential to be ableto capture a user's unique elements and features, and be able tocontinuously adapt the training method to the user's evolving skill.

Skilled behavior relies on organized strategies and builds on thewell-defined hierarchical organization of neurological processes.Therefore, to enable the systematic process, going from the assessmentand description of skill to the synthesis of feedback to the user, it isnecessary to define a modeling language that captures the structure andorganization of movement and is grounded on the fundamental principlesand principles of human movement science.

Following this language metaphor, conceptually, the core technologyfocuses on decoding movement data to extract relevant movement elementsthat can be used for skill analysis. The relevant elements in naturalspeech processing are the units of organization of speech production,known as phonemes. The decoded phonemes can then be used to identifywords and eventually the meaning of a sound bite. To help extractmovement units that are useful for skill analysis and diagnosis of anindividual's movement technique, these units are related to the processused for movement production. This analysis can then be translated intoinstructions and synthesize augmentation systems.

In parallel, the system utilizes an infrastructure to operationalize thevarious processes. The basis of the infrastructure is a data structurederived from the movement units that support efficient handling,processing, tracking, and managing of motion skill data. In addition,the structure allows codification of skill components and theirfunctional characteristics to design feedback mechanisms that targetprecise aspects of the movement skill performance and learning.

The proposed skill model and accompanying technology accommodates thenuances that naturally occur in human performance, and build on thestructural features inherent to the human movement system and itsvarious functional and learning mechanisms. Moreover, the methodscapture both the global skill components that give users its versatileperformance in an activity domain, and the specific skill componentsneeded for performance and adaptation to the specific task elements andconditions.

FIGS. 4A-4E show examples of movement architecture for the primarymovement unit for other movement activities (tennis movement unit 410,golf movement unit 420, rehabilitation movement unit 430, skiingmovement unit 440, running movement unit 450 and swimming movement unit450; e.g., for a user 10 of FIG. 2 ). The drawings also highlight themovement phases and the primary outcome.

The augmented skill platform is configurable to create an integratedenvironment for training, maintaining and rehabilitating motion skillsby combining motion capture technology, skill modeling and analysistools, and a set of feedback modalities that can target precise aspectsof movement performance. The system trains movement techniques tooptimize a set of outcomes that are relevant to the activity over itsdomain of operation.

Any task can be described by environment elements EE, and task elementsTE. For example, a person manipulates a device (e.g., tennis racket),end effector or piece of equipment, to interact with the task elementsTE (e.g. tennis ball). In addition, there may be miscellaneousaccessories Z such as shoes, clothing, that may be relevant for thedescription of the activity. The workspace W is contained in theenvironment and is specified by various constraints and rules thatcharacterize the task's success and performance (e.g. the tennis courtand tennis game).

In tennis, the person is the player (or players); the task environmentis the tennis court; the task element is the tennis ball; and theequipment is the tennis racket, and the accessories Z are the shoes andother pieces of attire such as an arm or head band. In addition, avariety of output devices can be included, including graphical displays(e.g. LCD, OLED, etc.), haptic devices (e.g. embedded in the racketgrip), speakers, etc. Finally, consider a variety of input devices,including, touch sensitive display (user interface), keyboard, etc. Theinput and output devices may be integrated in the form of a smart watch,tablet, or a wearable device that can be worn by the person.

The overall elements, agents and other components used, including themeasurement, input and output devices, are referred to as the augmentedhuman system or simply the system S. Other examples of systems that havethis general setup include a robotic system, a cybernetic system (e.g. ahuman fitted with a prosthetic), and a human-machine system (humanoperating a robot through tele-operation). For example, a roboticsurgical system such as the DAVINCI® Surgical System (available fromIntuitive Surgical, Inc.) is a robot that is an example of integratedaugmented movement skill system.

Measurements y can be obtained from different components of the humanactors, equipment, or system. Typically, instrumentation is designed toobtain measurements that encompass relevant variables for the particularlevel of analysis. For example, in the analysis of human tennis strokeperformance, the states, or a subset of the racket motion may besufficient. Additional measurements about the body segments (e.g. arm,legs, feet, etc.) are necessary to enable an analysis of the movement onthe court, the footwork, or the body motion such as the kinematic chainor other movement units.

These measurements can be obtained using a variety of technologies,including inertial motion unit (IMU), visual or optical trackingsystems, etc. Examples include the use of video cameras that capture thebroader agent behavior and the task environment. Vision processing canalso be used to extract information about the motion of individual bodysegments.

An important class of measurements are those that capture physiologicalquantities. For example, a gaze tracking system to measure the visualattention. Thus, as shown in FIG. 2 , a user 10 (or player) holding atennis racket 20 which impacts a ball 30 during the swing of the rackethas a gaze 12 which follows a trajectory which changes during themotion. One or more motion tracking cameras 210, 210′ can be providedwhich capture data related to the user 10, the tennis racket 20, theball 30, the motion and the environment. Inertial measurement units canbe embedded or affixed to the equipment; worn by the agent to measurethe movement of body segment; or even placed on the skin or implanted inthe body to measure muscle activity or neural signals involved in thecontrol of muscles.

See FIG. 5 , which illustrates an arm 14 of a user with a surfaceelectromyogram (EMG) sensor 510, an intramuscular electrode 520 and anIMU 530 for illustration purposes. The IMU measures velocity,orientation and gravitational forces using a combination ofaccelerometers and gyroscopes, and sometimes also magnetometers.

In addition to the measurements, data fusion and state estimationtechniques may be implemented to determine states x that are notdirectly measured. For example, in most applications using IMUs, theorientation of a body segment or piece of equipment requires an attitudeestimator which combines angular rate data from the gyroscopes, theaccelerations from the accelerometer and the magnetic field strengthfrom the magnetometer.

An example of data fusion and estimation is the use of extracted bodysegment or equipment motion information from a vision-based trackingalgorithm, applied to video data from video cameras, and IMU data from adevice on either the body segment or equipment. Such a data fusionsystem can be used to provide an accurate estimation of absolute pose ofbody segment or equipment. Typical cyber-physical systems are describedformally using hybrid system notation. This notation system combinescontinuous and discrete quantities.

For example, the movement of the agent may be governed by physical lawsthat result in nonlinear continuous time differential equations.Discrete variables may be used to evaluate conditions associated withspecific events, such as counting strokes in a tennis game or scoringthe game based on ball trajectory relative to the task environment andrules. Categories of state variables include: controlled variables,specific behavioral variables such as the visual gaze vector, andfeatures used as cues by the agent to make decisions.

Actions are typically taken by the user and represent the addition offorce or energy to the system. Actions are typically applied to specificlocations such as the end effector or equipment. Actions are oftenmotivated by a deliberate desire to achieve a particular outcome.

In tennis, for example, the player wants to impart a specific effect onthe ball (velocity and spin), with the ultimate goal of driving it to aspecific location on the opponent's court side. Events can be defined byparticular state conditions. For example, in tennis, a major event isthe impact of the ball on the racket. Events can be expressed formallyby constraints on the system states, e.g. racket acceleration exceedinga threshold due to the impact, or alternatively, the impact can bedetected when the ball and racket velocity are equal. Other relevantevents in tennis include contact of the ball with the ground and whenthe ball crosses the net.

Outcomes are defined as quantities that capture the relevantcharacteristics of the agent's behavior in a task performance. Toprovide a concise description, outcomes can be categorizedhierarchically, e.g. primary outcome, secondary outcomes, etc. Thedefinition of outcomes are a function of the scope and level of theanalysis. Expressed formally, outcomes are a subsect of the systemstates (e.g. at specific times, defined by events) or a function of thestates.

For example, in tennis, primary outcomes are the characteristicsassociated with the racket-ball impact, such as the spin of the ballwhen it leaves the racket or the ball's velocity. Secondary outcomescould include the location of the ball on the racket's string bed.Depending on the level of analysis (and available measurements), morecomprehensive outcomes include the location of the ball's impact on thecourt. The skill of an agent A is the effectiveness with which the agentis using its body and/or tool, equipment, etc., to achieve desired taskoutcomes TO and more generally interact with, and/or adapt to theenvironment elements EE and task elements TE.

Miscellaneous quantities include task or game rules (e.g. rules of thetennis game). Decision rules if one of the agent or its equipment oraccessory is computer controlled, e.g., control laws used to control aprosthetic limb, or for autonomous agents, the rules and algorithms thatspecify its behavior and actions in the environment and in response tothe environment and/or other agent(s).

FIG. 1 illustrates an overview of the system and is followed by adescription of the ‘augmented human system’, and finally, the generalmotion model, skill model, and the different augmentation modalities.The iterative training process illustrated in FIG. 1 illustrates twoprimary feedback loops: an assessment loop 100 tracks the skillacquisition process, and an augmentation loop 120 augments the user'shuman movement behavior during the training and task performance. Theassessment loop 100 can be used to track and update information aboutthe user's skills, including motion models and skill models, as well asdiagnostic tools used to identify specific deficiencies in movementtechnique.

The identified motion and skill models, combined with the diagnosticassessment provides the basis for generating a set of instructions,which are used to organize the training process, and synthesize cueinglaws used to drive the augmentation. A user receives two primary formsof feedback: instructions and real-time cues. The instructions aretypically generated during a session at particular intervals, e.g.completion of a training set, or after a training session. Instructionsare typically presented in visual form and emphasize more comprehensiveaspects of performance and skill.

The augmentation loop can be used to exercise movement by focusing byfocusing on movement characteristics that have been identified throughthe diagnostic tools. The cueing process targets specificcharacteristics to directly impact movement outcome and performance. Thecueing system computes feedback signals using algorithms that aresynthesized based on the motion and skill models derived during theassessment. These cues are communicated in real-time to the user. Theassessment and augmentation feedback are delivered following thehierarchical organization that takes into account the hierarchicalstructure of skill development and the temporal characteristics of themovement and skill attributes.

The training assessment loop is managed by a training agent. Theaugmentation loop is managed by a cueing agent. These agentsoperationalize the two processes and are able to track progress at thesetwo levels and provide the necessary user functionalities (see FIGS. 1and 6 ).

FIG. 6 illustrates the primary components of the proposed platform totarget skills 140. The platform depicts a human user 10 (or subject)having a movement system 16 (including a central nervous system and aperipheral nervous system) who exercises his or her movement skills in aparticular activity (tennis as shown here) in its corresponding taskenvironment. Some activities include a piece of equipment such as tennisracket 20 and a ball. Movements are used to either produce specificoutcomes that are relevant to the activity, or to adapt to environmentalconditions.

For example, in tennis, the primary outcome is the control of the balltrajectory (it encompasses characteristics such as impact location onthe court, the impact velocity, and spin effect determining how the ballwill bounce when it lands on the other end of the court). The user isinstrumented with some form of motion measurement device such as aninertial measurement unit (IMU) which acquires movement data 622. Themotion measurement devices can be distributed on relevant body segmentsand/or equipment. The user's motion can also be measured using motioncapture system such as a vision-based motion-tracking system. Inaddition, the system can also be configured to monitor the taskenvironment and track relevant elements, such as the motion of thetennis ball relative to the tennis court. Sensor devices can also beembedded in the ball or other relevant elements (e.g. instrumentedtennis ball).

The real-time feedback augmentation relies on cueing laws that areimplemented on a cueing system 624 which has a cue processor (see alsoFIG. 7 ). These cueing laws can provide continuous feedback, based onmovement profile characteristics, or intermittent feedback, based ondiscrete movement characteristics via cueing stimuli 626. The signalsfrom the cueing laws are communicated to the user using a cue generatorwhich is part of the cueing system 624, which produces cue stimuli thatcan be perceived by the user during movement performance. The cueingsystem 624 and generator can be integrated into wearable devices.

Typical cue stimuli include audible, visual, or haptic. The transducerscan be integrated in clothing, accessories, etc. The representation ofmovement skills requires breaking down motion data according to themultiple levels of organization of the movement system. If thisrepresentation proceeds based on fundamental properties of humanmovement system, the representation will provide assessment of skill ina grounded manner that will more easily and directly translate intotraining interventions.

A local host 628 (e.g., tablet, smartphone, smart watch, etc.) cancommunicate training and cueing information to the user. An augmentationsynthesis 630 can include training schedule or goal 632 and feedbackaugmentation 634. The augmentation synthesis 630 can communicatetraining instructions to the local host 628 or cueing laws to the cueingsystem 624. Data that is collected from a user via movement dataacquisition 622 can proceed through steps that include movementprocessing 610, which parses and classifies movement patterns resultingin phase segmentation, motion modeling 650, i.e., the movement patternrepertoire state machine, and skill assessment and diagnostic 660 whichprovides a repertoire of outcome derivatives. The data resulting frommovement processing, motion modeling and skill assessment, is thenprovided to the augmentation synthesis 630.

FIG. 7 provides additional detail about the operation of the cueprocessor 700. The cue processor 700 has three main operations: a phaseestimation and feature extraction 710, a cueing law 720 and a cuegeneration 730.

The phase estimation and feature extraction 710 further includes one ormore of a detect/predict phase initiation 712, an extract initial phasestate 714, an extract phase profile parameter 716, and an extractoutcome 718. The cueing law 720 further includes one or more ofcomparing with reference timing 722, comparing with target state values724, comparing with reference profile 726, and comparing with targetoutcome 728. The cue generation 730 further includes one or more ofphase transition cue 732, phase profile cue 734, alert 736, and outcomevalidation cue 738.

The input to the cue processor 700 is motion data while the output isone or more cue signals. A cue signal 750 is an output of the cueprocessor 700. The cue signal 750 has one or more of each of a phasetransition cue 752, a phase profile cue 754, an alert 756, and anoutcome validation 758.

FIG. 8 shows the overall temporal structure of movement activity for asession. The figure highlights the primary levels of delineationincluding session, sets and rallies (consecutive strokes), followed bythe primary unit of organization of movement, in this example thestroke, which is further decomposed into phases and finally actions.

The brain's way of organization and control of movement is fundamentallythe same for most skilled movement activities. The interaction of a userand task elements and environments results in specific movementarchitectures. The movement architecture and phase decomposition resultsfrom the need to accommodate the biological constraints and the type ofoutcomes that are specific to the task. However, the same general typeof levels of organization and associated movement elements. FIGS. 4A-4Eshow an example of primary movement patterns and phase decompositionacross different movement activities. Many activities share the samebasic movement structure and organization. Therefore, the motion modeland skill model, as well as skill analysis and augmentation can be usedover a broad range of activities.

Skilled movements can be modeled in terms of the three major levels ofmovement system organization:

-   -   (1) the global delineation of movement into distinct movement        patterns with their associated primary outcomes, which can        collectively be described in terms of the repertoire of        movements;    -   (2) the temporal delineation of the patterns into phase        segments; and    -   (3) the phases can subsequently be decomposed in movement        synergies.

The movement repertoire describes the collection of movement patternsused by a user to accomplish the range of outcomes necessary toefficiently deal with various environmental and task constraints in theparticular activity or adapt to the conditions. The development ofrepertoires is characteristic of open skills. Complex movements aretypically composed of distinct phases. Moreover, the movement phasestructure describes the user's movement technique delineating movementphase sequence developed to achieve the outcome associated with eachclass.

Movement or muscle synergies describe how the brain organizes movementto utilize the body's capabilities. The muscle or movement synergiesrepresent the basic building blocks or units of the movement behaviorassociated with the movement system. The synergies describe how muscleand body segments are coordinated to produce the movement phases thatcompose the movement patterns. The motion model provides the basis forthe analysis of technique and skills.

The motion classification and subsequent mapping describes the range ofmovement patterns acquired by an individual. Repertoire diversificationconveys information about the versatility of the player. Movementpatterns are typically associated with primary outcomes and therefore,the repertoire also provides a description of the range of outcomes.Movement patterns are dynamic, therefore, tracking their developmentprovides unique information about how an individual adapts to itsenvironment and task.

The phase decomposition of the different movement classes in therepertoire describes how an individual uses different technique for thedifferent outcomes in the task. The phase segment profiles hold detailedinformation about how individual users achieve their movement outcome.Since the phase structure results from the functional and biomechanicalconstraints, the analysis of the movement techniques in terms offeatures associated with the phases conveys information that is moredirectly relevant to training. At the same time this representation issparse compared to the entire time series of a movement pattern.Variability or differentiation of the profiles in the same class conveysinformation about how the users achieve their outcome and deal withchanges in conditions.

Movement phases, such as those found in tennis strokes, involve complexmovement coordination. The phase decomposition conveys information aboutthe ability of the user to exploit the natural movement biomechanics andtask elements and constraints. Trajectories of each phase are achievedthrough coordination of numerous muscles and body segments. Thedecomposition of movement phases allows to identify muscle synergies andbiomechanical characteristics. Finally, each phase also has specificmechanisms to achieve adaptive capabilities. For example, phases arepart of the sensory-motor structure used to perform under uncertain anddynamics conditions. Movement phases are an essential part of achievingproper timing to synchronize movements to external task and environmentelements. Phase are triggered by different sensory cues.

The system is configurable as an augmentation platform that helps trainor rehabilitate for a variety of movement applications. The elements ofthe movement structure and organization just described represent themain elements of a language or codification system that can be used tobuild an integrated movement processing, analysis and augmentationsystem.

FIG. 6 illustrates how these processes are deployed to enable augmentedhuman movement activity. The elements of the movement structure andorganization represent the main elements that are learned and stored bya human user. Therefore, through the effect of skill augmentation, theyrepresent the elements that get operated on through the effect offeedback augmentation. At the assessment loop level, the iterativetraining process effectively reorganizes the individual's repertoire. Atthe augmentation loop level, the feedback augmentation reshapes (orreprograms) the movement patterns by manipulating the movement phasecharacteristics and their relationship to the outcome.

The processes I-III are the motion processing functions which are partof the assessment loop. They support the training assessment and updatethe algorithms that support the augmentation (finite-state model forfinite-state estimation, cueing laws, etc.) and include movementprocessing 610, motion modeling 650, and skill assessment and diagnostic660. Processes IV-V are the synthesis and realization of augmentationsystem that support the augmentation loop. They are divided into twolevels: instructions and real-time feedback. The instructions aretypically implemented on a host equipped with a screen or other means todisplay symbolic or visual information. The feedback cue is implementedin real-time through the cue processor and generator and is based onsignals and cues (see FIGS. 7 and 10 ).

FIG. 9 illustrates the system integration between the movementcomponents that are extracted from the measurement data and highlightsthe synergy between the components and processes used for skillassessment and diagnosis and synthesis of various feedback elements. Themovement patterns of interest are those that are associated with theprimary outcomes for the given activity. They are called motionprimitives or primary movement units (MU).

The motion skill processing components include movement processing data910, movement model 950, skill model assessment and diagnosis 960, andfeedback synthesis 930. The system input is motion data. The movementdata processing 910 includes, detecting, parsing and extracting movementpatterns and outcomes 918, classifying movement patterns 916, segmentingmovement phases 914, and decomposing movement phase and muscle synergy912. The movement model 950 including a movement pattern repertoire 956of one or more pattern profiles, a pattern phase profile 954 of one ormore phase profiles, and a phase functional feature 952 which includesmovement biomechanics, neuro-motor performance, task constraints, etc.

The skill model and assessment diagnosis 960 includes a repertoire 962(e.g. repertoire maps, repertoire metrics (such as size andvariability), and outcomes), phase profile 964 (e.g., profilecharacteristics, performance measures and optimality measure), andfunctional 966 (e.g. phase profile outcome derivatives, phase transitionoutcome derivatives, movement adaptation). Feedback synthesis 930includes instructions 932 (e.g., data for reports, training protocols),validation alerts 934 (e.g., reinforcement, outcome validation,performance cues, phase alerts), and cueing laws 936 (e.g., real-timefeedback, phase reinforcement, outcome feature validation, outcomeoptimization, phase timing).

Movement measurement data is collected during training, matches orregular performance. This data is processed to extract movementpatterns, or primary movement units, relevant to the activity. Themotion data is parsed to extract the segments associated with theprimary actions and extract information associated with their outcomes.The patterns are classified to extract different categories and classesof motion patterns (see FIG. 11 ).

The movement classes and outcomes form the movement repertoire. Movementpatterns are decomposed following the hierarchical organization,including the segmentation of the motion phase, and decomposition of thephase segments according to movement synergies.

The different types of primary movement patterns from the movementrepertoire and their phase and synergy components collectively form themovement model (see 950 in FIG. 9 ). The movement model provides datawhich can be used for skill analysis and diagnosis (see 960 in FIG. 9 ).Skill analysis and diagnosis encompasses the three levels of themovement model in FIG. 9 .

At the repertoire level, information about the range of patterns andoutcomes can be assessed relative to the task requirements. At themovement phase level, information about the movement technique and itsrelationship with the performance and outcome. At the synergy level, themovement components describe relationships between the movement phaseprofiles and the coordination of body segments and muscles. The proposedframework, based on the organization of movement in terms of the motionmodel and skill model shown in FIG. 9 , makes it possible to organizemovement assessment and feedback cueing in a systematic fashion.

An assessment loop tracks the user's longitudinal skill development. Atraining agent then provides the environment to view and manage trainingassessments and instructions. Based on the assessments, the trainingagent provides various outputs to the user to help understand and trainmovement technique.

The two primary outputs are instructions and feedback augmentations. Theinstructions operate at the knowledge level and are best suited todocument medium to long-term aspects in skill development. The feedbackaugmentation operates at the signal and cue level and is effective inreal-time, during performance.

Feedback augmentation synthesis 630 shown in FIG. 6 translates resultsof the analysis and diagnostics into training instructions which areprovided to the user during augmentation 620. Examples of the synthesisare provided in the detailed description and the tennis example.Realization refers to the process of translating and communicatinginstructions into an understandable and actionable form which ispresented to the user during augmentation 620. The instructions can begenerated at different times: during performance, immediately followinga movement, or after the training session. The modalities ofcommunication include symbolic language (spoken, text), cues (visual,audible, haptic) or signals (visual stimuli, audible stimuli, hapticstimuli). The modalities of communication are implemented according tohumans' information processing and functional properties associated withthe structure and organization of movement.

The instructions can be organized into three levels:

-   -   (1) Reinforces the optimal movement features for a particular        movement class with feedback or cueing law. The cueing laws        account for specific phases in movement pattern segments to        perceptual and sensory-motor processes.    -   (2) Defines “training schedule” based on the organization of the        movement repertoire, building the repertoire according to the        hierarchical relationships between movement patterns and their        associated phase segmentation.    -   (3) Expands the movement repertoire using an individual's        existing movement patterns.

The feedback augmentation is set up to generate stimuli that reinforcesthe features in the movement technique that lead to outcome improvement.This principle is based on learning theory, such as the Theory ofNeuronal Group Selection. The theory essentially states that thebehaviors that are associated with good outcomes are selected andreinforced, while those that are detrimental to the outcomes areeliminated.

The movement phase structure is also associated with specific functionalmechanisms used for adaptation to changing conditions, such as timingand strength. The functional characteristics of motor programs and theassociated sensory and adaptation mechanisms provide a basis fortraining the larger capabilities such as perceptual mechanisms neededfor task-level interactions (environment and task elements).

The results of the assessment are presented through the communicationsystem. The training and cueing agent are the systems used to manage theassessment loop and augmentation loop (see FIG. 1 ). The technology fortracking and training movement skills can employ various informationprocessing and management capabilities.

FIG. 13 illustrates an overview of a general platform architecture thatorganizes and manages a user's movement skills. The platform has adevelopment algorithm server 1310 which can be stored in a cloudcomputing environment, and a motion and applications server 1320, whichcan also be stored in a cloud computing environment.

A host 1340 can be provided which is, for example, a smart watch, smartphone or tablet, which includes a user interface and training agentalong with a device manager. Additionally, embedded devices or sensorscan be associated with a piece of equipment (such as a tennis racket 20,skies, or skateboard) associated with the activity (if any).

The platform combines components used for communication, userinterfacing, computing and data storage. In particular, it combinescloud computing, which makes it possible to store, manage and processlarge amounts of data, with embedded sensing and computing, which makesit possible to build the necessary wearable hardware for data collectionand real time feedback. Such a system opens opportunities forcapabilities that go well beyond what can be achieved with data from asingle individual.

Data from a population of players opens the possibility to analyzeskills from different skill levels and differences in biomechanics, age,health, equipment, etc. Therefore, an important aspect of the platformis the organization and management of the collective motion data, motionmodels, skill models, cueing laws, etc.

A capability enabled by population data is the availability ofinformation about movement technique at different skill levels. This canbe used to cross-reference the skill analysis of similarly skilledindividuals provides opportunities to borrow technical advantages fromone individual to enhance training of a second individual. As will beappreciated by those skilled in the art, the use of a cloud computingenvironment is just one configuration and other computing environmentscan be used without departing from the scope of the disclosure.

To enable such large-scale organization and management capabilities,additional levels and control features can be added to the components inFIG. 1 .

The movement analysis and cueing and augmentation platform has aplurality of functions, including, for example (as shown in FIG. 9 ):

-   -   (i) motion processing function,    -   (ii) motion model and assessment functions,    -   (iii) feedback augmentation functions, and    -   (iv) skill development tracking functions.

The four subcomponents enable: parsing, classification, phasesegmentation and decomposition of the movement into synergies. Parsingis the process of segmenting and extracting the data segment that isassociated with primary movement patterns, or movement data processing910. These patterns are the primary movement units in the organizationof movement behavior used to accomplish the various primary outcomes inan activity (e.g. stroke in tennis, carving turn in skiing).

For example, in tennis, the primary movement outcome corresponds to theracket-ball impact (see FIGS. 4A-4E). Outcomes have distinct featuresthat can be detected. Therefore, for goal-directed movement behaviors,parsing can be performed by detecting features associated with themovement outcome. Reaching movements or other goal-directed movements,as well as throwing or intercepting movements, can be identifiedsimilarly. Parsing can also be accomplished by using more generalfeatures associated with movement patterns. Other features include, forexample, kinematic characteristics such as racket acceleration in theforward swing phase of a tennis stroke.

For non-goal directed movements, such as skiing, features of themovement maneuver typically provide sufficient information to parse thisbehavior into a sequence of movement primitives. For example, in skiing,the turning maneuvers can be detected from kinematics of the apex or theentry phase of the turn. Outcomes are defined based on the activity, andare typically known quantities. In tennis, for example, the outcomes ofa stroke are the ball's topspin, speed, direction, etc. The outcomes canbe defined at different levels, such as the stroke level and the gamelevel.

In tennis, the latter is related to the game structure and scoringscheme. Some outcomes can be directly determined from the measurements,while other need to be estimated from the measurements. Therefore, atthe functional level, parsing involves: Discovering features to detectthe event(s) associated with MU; Generating a time series description ofthe MU; Extracting movement outcomes associated with the MU.

The system has inputs which include data streams, and outputs whichinclude time-series descriptions of MUs, their associated features, andthe outcome descriptors. Movement skill analysis applications requiresampling measurement data at rates significantly higher (100 Hz orhigher) compared to activity detection applications (10-25 Hz).

Motion can also involve multi-dimensional state variables. Therefore,the measurement time histories result in large quantities of data.Moreover, the measurement data is often expanded to include estimatesfor unmeasured quantities, such as equipment orientation in space. Thesituation is further complicated when multiple sensors attached todifferent body segments. The large data can be challenging to manipulateand analyze, especially for real-time applications.

Low-dimensional abstractions are used for efficient, real-timeclassification. A general approach to classification is to represent thetime series through low-dimensional feature vectors. These vectors canthen be used to determine movement classes efficiently using dataclustering techniques. The movement class-membership information, inturn, can then be used to reconstitute the full-dimensional movementclasses in terms of the original time histories. These can then be usedto perform movement analysis, such as for evaluating movement techniquesor skills.

FIG. 14 illustrates an overall classification of the motion patternprofiles. As used in the tennis example, starting from the extraction ofstroke features from the movement profiles, and subsequently their useto assign membership of the strokes to the various classes. An ensembleof movement profiles 1410 is generated, which has input into a movementprofile, membership assignment 1420, which in turn has input into anensemble of classified movement profiles 1430. Movement ensemble featurerepresentation 1440, provides input into movement profile classification1460, which in turn provides input to classified features 1480. Theclassified features provide input into the movement profile, membershipassignment 1420.

The measurement data is typically provided as time series y(t), t∈

. A typical MEMS IMU the measurements typically include the three linearaccelerations [a_(x); a_(y); a_(z)] and the three angular rates [p; q;r]. The measurements are usually down sampled (period T_(s)) andfiltered to yield time histories with appropriate information and size.Most applications involve some form of state estimation (e.g. attitudeestimation). Therefore, the classification is typically implemented onstate vector data that allows reliable descriptions of the movementperformance. The following example assumes the racket movement isdescribed by body acceleration and angular rate.

$\begin{matrix}{{X(t)} = \begin{bmatrix}{a_{x}(t)} & {a_{x}\left( {t + {kT}_{s}} \right)} & {a_{x}\left( {t + {2{kT}_{s}}} \right)} & \ldots & {a_{x}\left( {t + {NkT}_{s}} \right)} \\{a_{y}(t)} & {a_{y}\left( {t + {kT}_{s}} \right)} & {a_{y}\left( {t + {2{kT}_{s}}} \right)} & \ldots & {a_{y}\left( {t + {NkT}_{s}} \right)} \\{a_{z}(t)} & {a_{z}\left( {t + {kT}_{s}} \right)} & {a_{z}\left( {t + {2{kT}_{s}}} \right)} & \ldots & {a_{z}\left( {t + {NkT}_{s}} \right)} \\{p(t)} & {p\left( {t + {kT}_{s}} \right)} & {p\left( {t + {2{kT}_{s}}} \right)} & \ldots & {p\left( {t + {NkT}_{s}} \right)} \\{q(t)} & {q\left( {t + {kT}_{s}} \right)} & {q\left( {t + {2{kT}_{s}}} \right)} & \ldots & {q\left( {t + {NkT}_{s}} \right)} \\{r(t)} & {r\left( {t + {kT}_{s}} \right)} & {r\left( {t + {2{kT}_{s}}} \right)} & \ldots & {r\left( {t + {NkT}_{s}} \right)}\end{bmatrix}} & {{EQ}.2}\end{matrix}$

X is the n×N matrix, where n is the number of states and N=t_(meas)T_(s)is the number of samples. For example, a movement profile described byt_(meas)=2 second time history, sampled at 1 kHz result in 6×2000=12,000data samples.

Feature representation involves mapping the movement instances, X(t)∈

to their feature description F:

f _(i) : |X(t)∈

  EQ. 2.5

where

is the instance space and

is the feature space.

The features are selected to describe the movement profiles X(t)∈

using a small quantity of data, making efficient processing,classification and analysis possible. FIG. 14 illustrates the mappingfrom the original data to the feature space, classification and mappingthe original data into the class membership. The movement profilescollected in each class can then be used for detailed analysis of themovement technique and skill.

Features represent attributes of the time histories that make itpossible to distinguish between different movement types. In thefollowing, the features are defined to provide information about thekinematic and dynamic states at selected times over the length of amovement profile. The features can also be selected through more formalapproaches, such as based on analyzing ensembles of movement profilesand determining variant and invariant characteristics.

The variable characteristics, if they are not due to noise, representfeatures that will be best distinguished between movement types.Conversely, invariant characteristics carry no information aboutmovement classes. One approach to determine discriminative features isto perform principal component analysis (PCA) on an ensemble of movementprofiles. The first eigenvectors, associated with the largereigenvalues, describe the variance in the data, and hence can helpidentify data samples that are most informative.

Once a set of features that sufficiently describe a particular type ofmovement have been defined, these features, or a subset of them, canthen be used for classification. For basic classification, features canbe selected based on physical or empirical criteria related to themovement mechanics. Alternatively, the features can be selected based on“data-driven” processes such as PCA based feature analysis or K-Meansclustering.

For identifying tennis forehand/backhand or slice/topspin, it ispossible to determine necessary conditions for these movement types andoutcomes. The empirical classification yields a set of pre-definedstroke classes such as shown in the stroke tree (FIG. 11 ). In contrast,“data driven” classification provides information about the particularstructure of an individual's stroke repertoire. Its effectivenessdepends on the amount of movement data collected and how well anindividual's data spans the players' repertoire. However, individualdifferences in biomechanics (due to anatomy, injuries, illness, aging,etc.) can result in a repertoire that does not span the typical range ofstates.

The advantage of predefined classes, e.g. based on domain standards fora task or activity, is that they provide absolute criteria that areuseful when comparing multiple players. Predefined classes have theadvantage that they are absolute. Individuals can be compared based onthe same criteria and also provide a way to assess how well thecollected movement data spans a standard repertoire. The classificationplatform can be configured to combine both approaches. These couldrepresent standardized movement classification criteria for a particularactivity.

Data-driven classification would be used to gain insights intoindividual movement patterns. The latter could then be used to defineand refine standard class models. This type of analysis corresponds totop-level processes in the platform (shown as skill models and motionmodels library in the development algorithm server 1310 in FIG. 13 ).Given the predefined set of features, classification most often onlyrequires a subset of features. Feature analysis and selection isconcerned with identifying which subset of features to use from theavailable features. A common approach for feature analysis and selectionis principal component analysis (PCA).

The movement architecture can be modeled as a sequence of dynamics withinitial and goal sets. Therefore, the goal is to be able to segment thelarger movement units (e.g. the entire tennis stroke) into thesesubunits or phases as shown in FIG. 11 . The phase elements, which canbe described by individual phase dynamics F and associated initial andsub-goal state sets

_(i0) and

_(i1), respectively, emerge from the interaction of the biomechanicalsystem with the task environment and constraints (see e.g. the exampleof reaching motion).

The movement phases typically reflect the biomechanical constraints, aswell as factors associated with the dynamics, motor-control andinteractions with the environment and task (see FIG. 2 ). Capturing themovement phase structure makes it possible to better characterize thesecontributions. Finally, the insights gained from the phase decompositionare used to derive a finite-state model which can be used in thefinite-state movement phase estimation. This model describes the overallmotion as series of transitions between a finite number of states whichrepresent the motion phases.

Movement phases result from the biomechanical and neuro-motor system aswell as functional aspects related to the interactions with the task andenvironment elements. Further analysis of the movements within thephases can provide detailed information about the physical movementimplementation. For example, a tennis stroke combines the body,shoulder, elbow, and wrist motion. A beginner's stroke is mostlyachieved by a simple shoulder motion (horizontal abduction andextension). As the player improves, this basic motion combines withelbow and wrist movements to result in a more complex,higher-dimensional movement unit.

Movement synergies are determined from decomposition techniques such asnon-negative matrix factorization. The synergies that are determinedthrough mathematical decomposition techniques, such as principlecomponents or matrix factorization, are not necessarily related tomuscle activations. Nevertheless, these elements can still provideinsights into relevant spatio-temporal characteristics of the movement.This understanding can then be used to characterize skill and feedbackaugmentation.

Synergies can be extracted from a variety of measurements or states,such as the spatial configuration of relevant body segments. The type ofinsights and understanding that can be derived from synergies depends onthe information content of the movement measurements. To obtain musclesynergies that are representative of actual muscle activation patterns,EMG measurements or measurement of the nerve signals are usuallyrequired (e.g. see forearm pronation and supination in FIG. 5 ).Decompositions based on factorization technique still provide valuableinformation, especially if the results are connected with thebiomechanical analysis. Such analysis can be conducted for the primarybody segments involved in the phases.

The motion model (see FIG. 9 box II) is derived from the elements thatare extracted in movement processing. These elements are derived fromthe three levels of movement organization. Classification of MU intoseparate categories. The collection of classes forms the movementrepertoire (MR) associated with the user's experience and skill.Segmentation of MU into a sequence of phases and specific phase featuresthat are associated with the phases. Decomposition of the individualsegments or phases. The components are related to the muscle synergiesor patterns of activation underlying the body segment coordination. Thedecomposition can be used to relate the movement characteristics and themusculoskeletal constraints.

This collection of elements is incorporated into the motion model 150and skill model which can be included in the motion model 150 of FIG. 1. The motion model 150 can have a variety of inputs and outputs,including, for example:

-   -   Inputs: Collection of time series representation of the MU; and    -   Outputs: MU organization as data tree, with two levels: the MU        types and the phases for each MU type; classification features.

Skill can be determined by how movements utilize the natural movementcapabilities to accomplish their specific goals or outcomes, and alsohow these movements achieve adaptive capabilities to compensate forvarious conditions. The following describes how the movement model (seeFIG. 9 box II) can be used to assess skills and target skillaugmentation and cueing (see FIG. 9 box III).

Motion skill is traditionally assessed in terms of two aspects:knowledge of performance (KP) and knowledge of results (KR). The keyquestion for skill is how these two aspects are related, that is, howmovement technique can be used to achieve desired outcome. The stroketechnique has to satisfy a variety of constraints including thebiomechanical and task constraints, and also requires adapting forchanging task conditions. The movement repertoire describes the variouspatterns used by a user to accomplish the outcomes in the particularactivity or adapt to the conditions. The movement phase structuredescribes the user's movement technique, delineating the movement phasesequence developed to achieve the outcome associated with each class.The movement repertoire formed by those classes for a particularactivity domain describes the collection of movement patterns that havebeen acquired to generate the outcomes necessary to efficiently dealwith various environment and task constraints.

Movement or muscle synergies describe how the brain organizes movementto utilize the body's capabilities. The muscle or movement synergiesrepresent the basic building blocks or units of the movement behaviorassociated with the movement system. The synergies describe how thebrain coordinates muscles and body segments to produce the movementphases that compose the movement patterns. The user's skills can beassessed based on the characteristics of the repertoire, e.g., howbroadly they span behaviors, with measured or estimated outcomes. Themotion model captures movement elements and characteristics that reflectthe relationship between the task requirements and movementbiomechanics. These elements and characteristics can be used to achievea comprehensive assessment of movement skill and skill acquisition. Theapproach to this assessment is based on the movement structure andorganization and skill development principles.

The component of the movement skill assessment are shown in FIG. 9 (boxIII) and include:

-   -   (a) At the repertoire level the assessment focuses on the        general characteristics of the movement patterns and associated        outcomes and how those are organized in relationship to task and        environment requirements and elements.    -   (b) At the movement phase profile level, the assessment focuses        on the movement technique and patterning of motion.    -   (c) At the functional level, the assessment focuses on the        internal structure of the movement and its detailed functional        characteristics including the relationship between technique and        outcome, as well the mechanisms used for adaptation to task        conditions.

The repertoire represents a discretization of the behavioral space atthe task performance level. It reflects how the entire range of behavioris broken up to achieve outcomes under different task conditions whilebest exploiting the biomechanical, neuro-motor, perceptual and cognitivecapabilities. An individual's capacity to exploit the range ofconditions, and in turn create distinct categories, is also a reflectionof the skill level. The fact that the development of a repertoirerequires making categories, which involves recognizing environmental andtask conditions, etc. is also a result of the higher level cognitiveskills.

The different movement patterns that compose the repertoire representthe basic unit of organization associated with the concept of motorequivalences. The extracted classes measure the breadth of the motorrepertoire associated with a particular activity as well as the qualityof the synergies or movement classes. Given an activity, the goal is tospan a large enough repertoire to support versatile performance, i.e.,deal with different environmental and task constraints. The patternsthat span the repertoire should provide adaptation to uncertainties anddisturbances. It is expected that advanced players' repertoires arestructured and diversified with a broad range of stroke classes thathave distinct characteristics to achieve a range of outcomes. On theother hand, novice players' repertoires are expected to be much lessstructured with fewer patterns that are themselves less organized.Therefore, the amount of structure and the breadth of patterns and howwell they achieve their outcome provide basic elements of systematicskill evaluation.

Movement patterning and specific phase structure is a function of skilllevel. For example, beginners will adapt existing movement synergies toaccomplish the task. As they become more proficient, the number andtypes of movement phases, as well as the performance in each phase, willchange. For example, a beginner tennis player will first use rudimentarystroke techniques that primarily build from a forward swing phase and abackward swing phase. With more experience and training, the strokepatterning will refine to include more phases such as a follow throughto optimize the performance after the impact, at recovery, as well as amore sophisticated backswing with a back loop that makes it possible toachieve faster reaction and better satisfy the requirements for theperformance of the forward swing leading to the impact.

In contrast to the more global significance of the movement repertoire,each class represents a movement pattern that is acquired to providespecific adaptive performance for a particular environment, taskconstraint and outcome. While the motion classes in a repertoire spanthe broad range of conditions, movements within a class requireadaptation mechanisms to accommodate changes in task and environmentalconditions. The remaining component of variability in a given class,i.e. within motion profiles, is due to motor noise (e.g. ϵ_(t)).Therefore, once movement patterns have been classified, the analysis ofthe movement profile characteristics within each class provides detailedinformation pertaining to the individual's technique and skills (seephase segmentation).

For advanced players, the variability displayed within a class isprimarily associated with the adaptation to the task requirements suchas impact conditions and body configuration. For beginner players, thevariability is primarily from motor noise such as due to untrained orpoorly coordinated movement patterns. For beginner players, variabilityalso arises from poorly differentiated movement pattern classes, i.e.the user mixes two patterns without a sufficient definition of thecategory. For example, a user may adapt the same stroke structure toperform topspin and flat shots.

Predictable movement outcome requires precise configuration of bodysegments during phase transitions and reliable, repeatable pattern ofmuscle activation and body segment coordination during phase profile. Inaddition, various correction mechanisms are provided to correct foreffects of disturbances or uncertainties at different stages of themovement execution.

Movement phases break up the trajectory in a way that allows to bestexploit human musculoskeletal and neuro-motor capabilities given taskrequirements and environment constraints. The phases, therefore, oftencoincide with important features of the bio-mechanical, motor control ortask constraints.

The movement patterns once classified represent movements that areequivalent (in the classic sense described elsewhere). Therefore, thesetrajectories provide precise insights into the various functionalaspects related to the movement constraints, including the sensory-motormechanisms and bio-mechanics. Furthermore, movement skills, inparticular open skills result from environment interactions.

For example, in tennis, the strokes are determined by the outcome, whichis selected based on context such opportunities afforded by the game,and the continuous adaptation to the conditions, including the exacttrajectory of the oncoming ball and the movement of the opponent. Thestroke architecture is developed to accommodate the task interactionand, in particular, the coordination between information (perception)and movement execution. Therefore, phase decomposition providesinformation to evaluate how an individual adapts to the conditions andoptimizes the outcome. The movement model and associated decompositiontechnique and data structures can be exploited to generate variousoutputs that support analysis, visualization and feedback augmentation.

Movement skills diagnostics is aimed at generating actionableinformation that can be easily used to support the synthesis of traininginterventions and augmentations. The detailed motion characteristicsassociated with each phase makes it possible to analyze functionalcharacteristics that have been acquired to attain the outcomes, as wellas the mechanisms used for the adaptation to task conditions. Inparticular, once the movement technique and outcomes have been capturedin a concise and sufficiently descriptive form, this information can becombined to analyze the relationship between specific movement featuresand the movement outcomes.

This sensitivity analysis provides the basis for the synthesis offeedback augmentation laws. The general approach is to evaluatesfeatures of the movement technique, associated with the MU, thatcontribute to movements' outcomes and other attributes such as themovement's adaptive capabilities. These include:

-   -   (1) Identify movement features for a given movement class that        contribute most to the outcome.    -   (2) Identify movement features for a given movement class        associated with an individual's best outcomes. These features        define the optimal movement features for the particular movement        class and outcome.    -   (3) Identify movement features for a given movement class        associated with adaptation.

Variability in technique of a movement class for a given outcome isassociated with adjustments in technique used to compensate for taskconditions. Inputs can include motion class; parameters can includeoutcome or performance criteria; and outputs can include feature vector.

Understanding how feedback can be designed so that it targets specificaspects of the movement techniques and guides the acquisition of skillin a formal analytical framework is needed. In engineering applications,dynamic programing can be used to determine trajectory x(t) to optimizea cost functional J(x(t)) which could be the movement outcome or otheraspects of movement performance and results, such as the energy of themovement (i.e., the smoothness or jerkiness of movement). Movementdescribed by a sequence of phases allows to parameterize movement with amuch smaller number of variables than would otherwise be necessary. Theset of parameters for the proposed movement model includes the movementphase initial and terminal conditions, timing information and theparameters of the phase profile dynamics F_(i). Profile dynamics are theresult of the force field that are produced by the muscle coordinationpatterns or muscle synergies.

The representation in terms of movement phases also helps betterestablish a relationship between the outcome and movementcharacteristics. Namely, the optimization of the outcome can beformulated as a parametric optimization problem, or parametricprogramming. This is possible because the outcome can be expressed as afunction of a finite number of parameters instead of the entire movementtrajectory. This model provides a basis for both the functional analysisof movement and the formulation of the feedback augmentation.

The formulation of the movement optimization as a parametricoptimization problem assumes that the movement outcome (or performance)can be expressed as an objective or utility function J(x, Θ) that needsto be maximized (or minimized). The quantities x is the optimizationvariables and Θ are the parameters. For the movement optimizationformulation, Θ can be used to represent the structure of the movementpattern, i.e., movement pattern configuration parameters, and x areparameters specifying the movement profile within the general pattern,i.e., movement profile features.

For example, given a tennis stroke class, e.g., a forehand top spin, theconfiguration parameters determine the stroke pattern that is associatedwith the movement structure associated with the individual's stroke (seeFIGS. 15A-15E). The movement profile parameters include perturbationswithin the movement structure such as perturbations in the movementstate at phase transitions and/or perturbations about the phaseprofiles.

The goal of optimization is to determine parameter values that minimizeor maximize the objective function. A constraint function g(x, Θ)≤0 canbe included to describe feasible ranges of movement configuration, aswell as a parameter space Θ∈Ξ to describe the range of values of themovement configuration parameters. A simple solution to the parametricoptimization is the gradient descent (or ascent depending on if the goalis to minimize or maximize the objective function). Given a movementpattern structure and profile feature for a trajectory instance,described by Θ and x, respectively, the objective function can beoptimized by iteratively updating the movement by applying smallvariations in the movement profile features: x′=x−γ∇_(x)J(x, Θ), where γis a step size.

This framework can be used to provide diagnostics of the movementtechnique, both at the profile feature level and the pattern level. Thegradient of the objective function with respect to the profile featuresx, for a movement structure specified by the movement patternconfiguration parameters Θ_(k), is given by:

∇_(x) J(x,Θ),Θ=Θ_(k),  EQ. 3

which provides information about the sensitivity of the objective oroutcome to profile features. Therefore, it is possible to use thisanalysis to identify the set of features that control the movementoutcome. In an example of spin imparted to a ball at impact changes theracket azimuth and elevation at back loop initiation and the elevationand roll rate at forward swing initiation (see FIG. 16B and FIG. 18B).

A similar approach can be used to determine pattern configurationcharacteristics and their impact on the outcome. The gradient of theobjective function with respect to the pattern configuration features Θ,given by:

∇_(Θ)(x,Θ),  EQ. 4

which provides information about the sensitivity of the objective oroutcome to pattern configuration features. However, perturbations inpattern structure may not be sufficiently continuous or smooth to allownumerical sensitivity analysis.

The skill model and diagnostic functions provide information that can betranslated into various augmentations (see 930 in FIG. 9 ). Theparametric programming framework demonstrates the basic elements neededfor skill diagnostics and synthesis of various forms of feedbackaugmentation. In practical implementations, the sensitivity analysis andeven model can be implemented using the statistical method from machinelearning. With this framework, it is possible to use this analysis toidentify the set of pattern features for movement outcome. Thisparticular information is useful for the diagnosis of movement patternand ultimately for designing feedback for reshaping the movement patternarchitecture.

The following describes the elements of the skill analysis used toenable a systematic assessment to provide necessary information tosynthesize augmentations. Movement data from a motion class in therepertoire and the associated outcome provide information about therange of variations in movement technique acquired by a subject. Thecomprehensive set of movement profiles can be sorted to determinedifferent subsets based on attributes such as energy, jerk, outcomelevel, etc. The features associated with a movement pattern and itsrange of outcome define the current feature envelope of that movementpattern. The envelope describes the functional characteristics withinthe movement structure and the variations and perturbations associatedwith the individual's movement performance. Some of the variation is dueto motor noise and some is associated with modulation of the movement tocontrol the outcome level and adapt to conditions.

For example, a subset of movement profiles can be identified that areoptimal for the subject's current movement technique, where optimalitycan be assessed using objective function such as energy. This subsetprovides a reference for the individual's best technique. Therefore,using movement profiles from this set can be used to reinforce movementtechnique. This can be achieved by providing the player or subject asignal that validates their best technique when it is detected duringthe movement performance.

Furthermore, within the same class of movement, the different profilesnaturally exhibit variations in the level of outcome. Capturing therelationship between movement features and the variations in outcomeprovides information about the technique used by the individual tomodulate the outcome. Similarly, the information about these featurescan be used to generate feedback signals to help the player understandthe outcome modulation.

Information within the set of movement can also provide the basis tohelp individuals further optimize their movement technique for exampleto improve smoothness or increase their level of outcome. Following thismodel, an individual's movement performance can be optimized bygenerating a signal that provides information about the direction theyshould modify the movement feature. For example, the parametersextracted from EQ. 3 can describe a set of profile features such as theconfiguration of the movement at the transitions points between phases,or a specific kinematic characteristic of the phase profiles.

Building on these concepts, a normal and extended envelope can bedefined. The envelope captures the features that are associated with themodulation of outcome (see FIG. 16D for the tennis example). Theextended envelope corresponds to the features that lead to increase inoutcome beyond the range currently achieved by the subject. The extendedenvelope and their associated features define the range of movementvariations that can be exploited for outcome optimization.

One of the challenges of training complex movement skills is the lack ofinformation about the performance or results. Few natural signals existto help a user know if their movement technique is correct. Thisknowledge is most effective when delivered during performance. Outcomevalidation feedback operates as real-time or near real-time assessment.Knowledge of performance (KR) and knowledge of results (KR) can help anindividual train. Yet, there isn't an automated system that can deliverfeedback signals automatically, in a systematic way, building on thehuman movement system and human skill acquisition process. The followingdescribes the type of feedback modalities and cueing mechanisms that canbe used to augment human performance.

The feedback is delivered within an augmentation system 620 depicted inFIG. 6 . The movement architecture and structure represent the mainelements that are learned and stored by a human user (see FIG. 6 ).Therefore, through the effect of the skill augmentation, they representthe elements that are manipulated through the effect of feedbackaugmentation. Which explains how feedback augmentation can be used toreprogram the human movement system. The feedback is conceived at twoprimary levels: At the augmentation loop level, the feedbackaugmentation reshapes (or reprograms) the movement patterns bymanipulating the movement phase characteristics and their relationshipto the outcome. The augmentation includes both cueing signals or stimuliprovided by a cueing system 624 and instructions provided by acommunication system 628. At the assessment loop level, the iterativetraining process effectively reorganizes the individual's repertoire.

Feedback synthesis refers to the process of translating informationabout the different skill assessment and diagnostic components (FIG. 9box II and III) into feedback at the different levels of theaugmentation hierarchy. The instructions can be generated at differenttimes: during performance, i.e. real-time cues, immediately following amovement (validation cues), or after the training session. Themodalities of communication include symbolic language (spoken, text),cues (visual, audible, haptic) or signals (visual stimuli, audible,haptic).

FIG. 21 illustrates a block diagram of the iterative training systemprocess. An overall performance database 2110 provides performance datato an overall skill assessment 2120. Training schedules can be generatedor updated 2130 which may be provided to a user profile. Additionally,training session management 2140 can incorporate information from thedatabase and other sensors to provide suggested training profiles whichare presented as training profile navigation 2150 to a user. The userselects a training profile and a training session is implemented 2160.Following training specific skill assessment 2170 can occur followed byan iterative repeat of the training session management 2140.Additionally, specific skill attributes database 2180 can provideinformation to the system.

The implementation of the augmentation follows the humans' informationprocessing and functional properties associated with the structure andorganization of movement as described. The augmentation modalities canalso be delineated based on knowledge, cues and signals nomenclature.Post session are typically in the form of visualizations orinstructions. They are considered knowledge level.

The real-time and near-real-time are typically based on signals and cues(see FIG. 22 ). FIG. 22 illustrates the control and informationprocessing hierarchy for human movement behaviors, highlighting theknowledge, rule-based and signal-based levels 2210, 2220 and 2230. FIG.2 also illustrates the proprioception and motor skill sensory components2232 and 2234, and the exteroception, low-level and high-levelperceptual components 2240, 2250, and 2260, used to relevant informationfrom the task environment 2270.

The primary feedback modalities are divided between real-time cues andpost session or set illustrated in FIG. 12 .

-   -   (1) Real-time feedback 1210 for reinforcing the optimal movement        features for a particular movement class. The cueing laws        account for specific phases in movement pattern segments to        perceptual and sensory-motor processes.    -   (2) Near real-time feedback 1220 provide information to the user        about the attributes and outcomes of the current movement.    -   (3) Post session feedback 1230 are long-term and include        “training schedule” based on the organization of the movement        repertoire, e.g. expanding the repertoire according to the        hierarchical relationships between movement patterns and their        associated phase segmentation.

The feedback cueing is based on the functional characteristicsassociated with the structure and organization of movement described bythe motion model. As described the finite-state model provides a sparserepresentation of the trajectory that is compatible with the humanmovement system. The movement architecture results from variousfunctional characteristics. This architecture therefore provides thebasic structure for the implementation of feedback augmentation.

From the finite-state model, the trajectory is represented by a sequenceof sub-trajectories x_(i), for each phase i and transition conditionsthat define the initial values for each phase. Following thefinite-state notation, features include states at the phase transitions(initial and terminal state x_(i0), x_(i1) shown in FIG. 3 and phaseprofile characteristics of the phase x_(i)(t), t_(i0)≤t≤t_(i1). Theformer is related to the initial and terminal (subgoal) conditions inthe movement trajectory in FIG. 3 . The latter is related to thedynamics, i.e., the force field characteristics associated with musclesynergies. These quantities are ostensibly the type of information thebrain encodes for the specification of motor programs.

The phase transition and phase profile structure defines two primarycueing mechanisms. Phase transition cues are cues that are directed atthe characteristic of the movement phase at the end of a phase andinitiation of the subsequent phase. The phase transitions act as a typeof subgoal, therefore, the phase transition cues can, for example,provide information about the intermediate movement configuration overthe overall movement pattern. Phase profile cues are cues that aredirected at the phase profile characteristics. These cues can helpachieve phase profile attributes, such as acceleration or angular rateprofile. Phase profile cues also affect the phase transition followingthe current phase profile.

These cueing mechanisms are used to achieve different cueing goals:validating or reinforcing aspects of the movement; alerting todiscourage particular movement features that are detrimental to theoutcome or body; and signaling to change movement technique such as forrefining the movement technique to maximize an outcome or maximize theeffectiveness of the movement technique for a given outcome level.

1) “Outcome validation” provides feedback signals to validate thesatisfactory achievement of the movement outcome. Outcome validationcues can be generated immediately following the movement. They act asreinforcement. This function is enabled by assessing the outcome at theinstant it is produced.

Consider an outcome z that can, for example, be expressed as a functionof the movement state:

Z=φ(x(t=t _(goal))),  EQ. 5

and consider the optimal outcome:

Z∈Z _(goal)*  EQ. 6

where Z_(goal)* represents the set of outcomes that are optimal (w.r.t.to an objective function J). It is possible to discretize the outcomeset to define outcome levels A, B, C:

Z _(goal) *={Z _(goal) ^(A) ,Z _(goal) ^(B) ,Z _(goal) ^(C)}  EQ. 7

With this formulation it is possible to define a feedback signal thatconveys information about the outcome level. For example, the outcometiers A, B, C can be mapped to tones which can be generated as audiopulses at the instant of movement outcome.

2) “Outcome feature validation” provides validation of the movementfeatures that are consistent with a given outcome level. The discretizedset of outcomes is mapped to the features at the phase transition (orphase profiles). For example, the phase profile at the movementinitiation for a level A outcome at phase transition i is given by:

x(t=t _(0i))∈X _(0i) ^(A)  EQ. 8

The regions can then be used to generate cues that reinforce themovement technique that is consistent with a desired outcome level. Thismakes it possible for the user to make associations between thebehavior, which is described by specific features of movement phasestructure, and the effect of these features on the outcome.

Outcome feature validation provides a feedback signal validating thatkey movement characteristics meet the required value to achieve thetarget outcome. This function is enabled by the movement phasedecomposition, which allows tracking the movement behavior acrossvarious phases and determine if they meet the conditions at the subgoallevels.

The phase transition and profile cues help users develop an awareness ofthe movement phase structure. The cueing provides information pertainingto both spatial coordination as well as temporal coordination. Theseforms of cueing therefore can also be regarded as types ofproprioceptive augmentation where the proprioceptive augmentationenhances awareness and conscious perception of the movementarchitecture.

3) “Alerts” can be conceived as the opposite of validation cues. Thisfeedback can be setup to have specific functional roles, such asalerting the user if the movement state trajectory is outside of thefeasible region g(x, Θ)≤0 (see FIG. 3 ). As with other cues, alertsemphasize conditions at phase transitions or during phases (phaseprofile). Alerts delivered concurrently with the movement features aremore effective than at times or attributes that are arbitrary.

An alert protecting users from injury or wear can be implemented basedon the analysis of the relationship between movement profiles and thebiomechanical system. For example, the phase profile during the forwardswing and impact can be mapped to the wrist and forearm joint motion andmuscle activity (FIG. 5 ). The synergy decomposition can be used todetermine joint deflections and muscle activation profiles. This mappingcan then be used to detect when these quantities are exceeding someacceptable values. Similarly, this information can be used to generatephase transition and phase profile cueing to help form movement patternsthat are compatible with an individual's musculoskeletal conditions.

4) “Outcome optimization” Following general movement skill acquisitionprinciples, once a basic movement pattern (or architecture) has beenacquired, it is refined and eventually optimized. As described,information available for a set of movement data can also provide thebasis to help individuals further optimize their movement technique.Optimization of the outcome corresponds to learning to modify themovement to increase the level of outcome. The movement is parameterizedby the set of features that influence the outcome. The set of featuresdefines the admissible trajectories. FIG. 3 shows the optimal goal state340 and the envelope of admissible trajectories 320. A particularcombination of features results in the best outcome level for thatparticular movement pattern.

Given a movement pattern architecture, specified by the finite-statemodel, the outcome can be defined by a function of the goal phase:

Z=φ(x(t=t _(goal)))=φ(x _(goal))  EQ. 9

The sensitivity of the outcome, with respect to the movement profile,can be described by a sensitivity function ∇_({tilde over (x)})φ(x, Θ)where x is the set of all phase features. The subset of features thathave the highest sensitivity define the control variables that can beused to cue the user to refine phase profiles. Similarly, thesensitivity of the outcome with respect to the movement patternarchitecture can be described by a sensitivity function∇_({tilde over (Θ)})φ(x, Θ) where Θ is the set of all patternarchitecture features. The subset of {tilde over (Θ)} that has thehighest sensitivity defines the control variables that can be used tocue the user to optimize pattern architecture.

Based on this analysis phase timing and initial state characteristicsare ideal targets (control variables) for cueing. This corresponds tocueing for the subgoal states (and initial states) of movement phases(FIG. 3 ). For example, in tennis, this would correspond to guiding thebackswing to reach optimal racket orientation (elevation, azimuth andpitch) at the onset of the forward swing.

5) “Movement Refinement”: Within an outcome level (EQ. 7), the movementcan be refined to achieve better efficiency, lower stress on the body,etc. The optimization of the movement for example to enhance efficiency,follows the same approach as for the outcome optimization but theoutcome level is fixed and the objective function captures the qualitiesthat are relevant to refinement such as jerk or energy.

6) “Pattern reshaping”: The idea is to extend previously formed patternsby inserting a new phase (see FIGS. 15A-15E). This is a form of patternrefinement that extends outside the movement pattern. This process canbe implemented by providing outcome feature validation cues thatreinforce new patterns. The cues operate according to the two differentforms of phase extensions: either inserting a new phase or breaking upan existing phase in two sub-phases.

The parametric optimization framework can be used for reshaping movementpattern architecture. For example, given the outcome function J(x, Θ),the gradient of this function with respect to the pattern configurationparameters Θ can help explore effects of variations in patternarchitecture. However, this formulation may be ill-posed since anexisting trajectory manifold for a user's particular motion patternclass will typically only have one pattern. However, as users progressin their skill acquisition, some elements of new patterns may beavailable in the data to support the data-driven analysis.Alternatively, simulation tools can be developed from the parameters ofuser's existing data to discover new patterns. Cueing laws can then bedesigned to reinforce features that will drive the pattern architectureto the refined architecture.

7) “Pattern formation”: The movement structure depends on skill leveland evolves with skill development.

Since the phase transition and phase profile cues only have limited totransform the movement structure and organization, the formation of newmovement patterns requires different cueing mechanisms.

The cueing formulation introduced earlier is adapted to the skill level,however, phase transition or phase profile cues may not providesufficient signals to help form new movement patterns. Therefore, it isnecessary to design feedback cues that can help form new movementarchitectures. The following describes cueing mechanisms that can beused to assist the formation and reshaping of movement patterning orphase architecture.

Complex movements, such as the tennis stroke, are dynamic processes,i.e., the temporal and spatial dimensions are coupled. Therefore, tomeet the impact conditions at a specific time and space and produceimpact conditions that lead to the desired outcome, the execution ofthese trajectories requires memory of the pattern spatial and stateconfiguration, which is given by the transition points, and memory ofthe muscle activation patterns (muscle synergies) that enable thedynamic execution of the movement phase profiles and transition betweenmovement phases.

Movement sequence learning theory shows that learning complex sequencesstarts with an explicit stage emphasizing spatial characteristicsfollowed by an implicit stage focused on the neuro-motor processaspects. The acquisition of spatial movement configuration is usuallyaided by direct demonstration or through the assistance of a visualdisplay. Once the new movement pattern's spatial configuration isassimilated, the subject has to learn the pattern of muscle activationand body segment coordination to successfully reproduce the movementpattern under dynamic conditions. This process is usually much lessconscious and relies on repetition to consolidate in the memory beforeit can be utilized reliably under typical task conditions.

Cueing mechanisms for outcome feature validation or refinement can helpconnect the explicit sequence elements and the implicit motor processes,and in turn, should help accelerate the formation and consolidation ofmovement patterns.

The real-time feedback augmentation using the cueing mechanisms areimplemented by a cueing system. The cueing system 1000 is composed of acueing processor 1010 and a cue generator 1030 as shown in FIG. 10 . Thecueing (cue) processor 1010 comprises a state machine 1020 and a cueinglaw 1012. The cue generator comprises a cue encoder 1032 and atransducer 1034. The cueing system 1000 can be integrated into astandalone device, such as an earpiece or a wearable device. The statemachine 1020 can be configured to provide a movement phase estimation1024 and a feature extraction 1022. The input of the cueing system 1000is motion data while the output is one or more cue stimulus (ifnecessary). Its primary functions can also be distributed across severalphysical systems; for example, the cue processor could be on theembedded device and the cue generator could be implemented on asmartwatch. The transducers used to produce the cue stimuli can beintegrated in the clothing or other accessories.

As the movement patterns unfold, the finite-state estimator identifiesthe movement phase and phase transitions. In addition, the knowledge ofthe movement state across phases makes it possible to extract associatedphase profiles and values of the movement state at transition. Based onthe knowledge of the skill assessment and skill model and diagnostic,the relevant phase and state characteristics are extracted (e.g. phaseoutcome derivatives, phase profile outcome derivatives and outcomes).The state estimator uses motion measurements to identify the movementphase with the corresponding initiation times and initial/terminalstates at phase transition. The phase estimate can be used to extractrelevant information about the movement phase, including, e.g., stateinformation and various derivatives that describe the relationshipbetween the movement characteristics and movement outcome. The cueinglaw uses this information to compute a cue signal. The cue signal isthen transformed into a stimulus by the cue generator.

The cueing system has four main components: phase state estimator,cueing law, cue encoder and transducer. The finite-state model is a keycomponent of the movement phase discrete state-estimation system.Estimation of the movement phase is essential for real-time feedback.Since there is latency in the response, the cues directed at phasetransition or phase profile characteristics should be based onprediction of the phase and relevant phase characteristics. The movementphases are identifiable through kinematic or dynamics features at phasetransition.

Formal detection rules can be formulated for these transitioncharacteristics. These can then be incorporated in estimation schemes,such as a finite-state estimator, that make it possible to automaticallysegment the movement into phases from the measurements. However, somemovements involve subtle structural features and the phase detectionrequires more sophisticated models. The movement phases can also bedetermined based on models of the state transition maps extracted fromnonlinear time series as discussed earlier. Phase state estimation isalso possible with statistical methods such as Hidden Markov Models(HMM).

The cueing law operates similarly to a feedback law. It takes stateinformation and computes a control signal to influence the behavior. Thecueing law is grounded in the skill model derived from the assessmentand diagnosis process, which identifies the movement characteristics fordesired movement performance and outcome. A primary example is the stateof the movement at phase transition and its influence on the movementoutcome discussed earlier. This relationship can be captured throughsensitivity analysis. The signals from the cueing laws are communicatedto the user using a cue generator, which produces cue stimuli that canbe perceived by the user during movement performance. Typical cuestimuli include audible, visual, or haptic stimuli. The cue generator iscomposed of a cue encoder, which transforms the cue signal into acodified cue signal, and a transducer, which transforms the cue signalinto a cue stimulus.

The purpose of the cue encoder is to create a cue stimulus from theinformation available in the cue signal that can easily be understood bythe user. The encoder is best designed based on the understanding of thetype of decoder and internal model that will be used by a user tointerpret the cue stimuli. In an audible cue, for example, cue stimulican be produced by using musical notes from a scale. The musical notesand harmonic principles represent an innate form of the internal modeland decoder for the interpretation of the cue stimuli. One way to encodecues is through quantization of the cue signals and assigning values tothe quantized signals. This allows the user to recognize differentlevels in the cue stimulus and associate these levels with relevantinformation. One approach is to use familiar stimuli codes, such asharmonic or color schemes (red, orange, green), etc. These forms arecommon enough that users will familiarize with the encoding scheme andprogressively more complex cues can be implemented.

The effectiveness of cueing depends on several characteristics includingthe perception of events, the perception of the structure of movement,and the timing characteristics in natural motor control. The lastcorrection before goal or impact has been estimated to take place atabout 290 msec (for visual feedback). That time depends on the feedbackmodality and is smaller for proprioceptive cues (about 100 msec) andaudible cues (about 150 msec). Therefore, little or no benefit isexpected for cueing the movement phase profile of phases that are withinshort time intervals of the primary movement goals. For example, thedata from the tennis experiments show little adaptation is visible inthe phase about 100 msec before the impact (see, e.g. FIG. 16A whichprovides an ensemble of racket angular rate profiles highlightingpatterns that can be used to identify stroke phases). Real-time cueing,may also require predicting phase transitions and the movement states atthe future transitions points.

To achieve systematic, data driven analysis and augmentation, it isnecessary to operationalize the training process. The skill augmentationcomponents are synthesized from the hierarchical motion and skill model.The elements needed to realize, track and manage augmented training areintroduced following the movement skill model.

The organizational and skill development model for addressing differentaspects of a user's skill components such as movement repertoiredevelopment and movement pattern phase formation can be broken into acortical level, subcortical level and spinal level. The cortical levelincludes movement repertoire (in the long term memory) and executive orhigh-level perception. The subcortical level includes movement phasepattern formation and reconfiguration, while the spinal level includesphysical implementation. These different levels can be targeted throughselection of feedback modalities.

The synthesis and realization of augmentation functions is delineated interms of the knowledge level and the cue/signal level. FIG. 1 depictsthe synthesis and realized components at those two levels. At theknowledge level, the skill diagnostics from the skill model are used tosynthesize training schedules and one or more instructions. At thesignal and cue level, they are used to synthesize cueing laws.

The training schedule is composed of training elements that specify thepatterns and outcomes that needs to be trained (for example refined oroptimized). Each training element can be performed under severalaugmentation modalities. These can be selected as user preferences.Feedback cues are packaged as cueing profile that target the cueinggoals (1)-(7) described earlier. Instructions summarize key insightsabout the individual's movement technique from the diagnostic results,encompassing the various levels of the assessment hierarchy.

In one implementation, the training schedules and instructions aremanaged by a training agent which provides expert system for managingthe long-term training goals. Similarly, the cueing profiles within thetraining schedules can be managed by a cueing agent, which providesassistance to the user on the selection of cues and tracks theireffectiveness. Skill assessments, in particular the outcome phase andprofile feature derivatives, provide measurements to determine the bestcombination of cueing modalities and mechanisms and how these modalitiesshould be used.

The cueing agent tracks the progress in movement technique and outcomefor a particular cueing profile and can recommend when to stop or switchto a different profile. In addition, the motion data acquired while acueing law is active provides further measurements for validation. Thecueing law effectiveness is assessed while it is active. Once progresswith a cueing profile has reached the expected level, predicted duringits synthesis, the training process returns to the training agent.

The structure of the finite-state model is expected to change overvarious stages of skill acquisition (see FIGS. 15A-15E). For example, abeginner primarily focuses on the forward swing and the impact (FIG.15A). The follow through (FIG. 15B) is a consequence of the impact andthe backswing is not integrated in the stroke and is essentially aloading movement. Over time the player learns to integrate the followthrough and backswing as part of a coherent stroke motion (FIG. 15C). Asusers extend their motion into more extensive regions of the statespace, additional bio-mechanical constraints and dynamical effectsbecome relevant and therefore need to be accounted for to exploit theextended envelope. As a result, the movement architecture can be quitedifferent and the finite-state model needs to be adapted to thesespecific characteristics such as the recovery and the back loop (FIGS.15D and 15E).

Therefore, the goal is to be able to identify the model structure fromthe data in order to account for individual differences due to techniqueand the effects of skill. For example, consider the tennis stroke. Inbeginners, early stroke movements are mostly undifferentiated forwardswing motions with the primary goal of intercepting the oncoming ball.

As tennis players continue playing, they begin to differentiate theirmotion to impart different effects on the ball (comparing strength andspin) and adjust to the impact conditions (comparing trajectory of theoncoming ball). They either discover these new stroke patterns bythemselves, through adaptation of their technique, and/or through thedirection of a coach. The stroke must be optimized as more specificoutcomes and demanding conditions are desired. For example, to achievestrong topspin effects, the stroke trajectory in the phases precedingthe impact (backswing and forward swing) must follow a profile that willlead to impact conditions that will generate topspin. For example, theracket head must drop sufficiently low before the forward swinginitiation.

The optimization of stroke techniques leads to specialization of themovement architecture. As a result of this movement refinement process,the overall movement structure evolves with skill. These characteristicssuggest that training schedules, as well as the entire movementanalysis, modeling, and feedback augmentation should proceediteratively, following the differentiation and specialization of themovement structure.

The states of the finite-state model can be conceived as a hierarchicalstructure that has primary states which are expanded into higher orderstates. For the tennis example, as shown in FIGS. 15A-15E, the mostelementary motion model is composed of three states: the stroke (forwardswing), the impact, and ready (or idle). Next, the stroke state can beexpanded to include follow through and/or backswing and so on. Thesethree states can then be further expanded: the forward stroke intostroke initiation and interception, the follow through into recoveryphases, and the backswing into backswing initiation and back loop. Notethat the phase structure, and hence the process of structure refinementis specific to a movement pattern.

The extension of the model follows two basic directions:

-   -   (a) addition or insertion of a new primary state such as the        case of delineating the swing into a backswing and forward        swing, and    -   (b) the refinement or expansion of an existing primary state        through the addition of a state such as the case with the        expansion of the backswing into a backswing and back loop        sub-phases.

The new primary states are shown in FIGS. 15A-15E in dashed lines andthe new expanded states are highlighted through a dashed linesurrounding these state. The distinction between the addition orexpansion is determined by hierarchical movement structure. Primarystates are movement sub-units while expansions are determined by factorsincluding movement control mechanisms. In some cases, these distinctionscan be ambiguous.

There is a relationship between the movement structure hierarchy and thenatural development of skills. Furthermore, primary states are common toa broader class of skills and techniques. For example, they may beshared by different stroke classes, or even be similar from one user toanother. Therefore, a low-order motion model can reliably describe abroad population of users. The higher-order models require more detailedunderstanding of an individual's technique. Therefore, the models can besynthesized through an incremental process, starting with the low-orderfinite-state model, which can then be refined as data becomes available.

The movement segmentation and finite-state models for the evolvingmovement architecture shown in FIGS. 15A-15E can be extracted fromindividual users in a data-driven fashion.

These models can be stored in a library and enable tracking of the skilldevelopment process as shown in FIG. 17 . With data from a sufficientlybroad skill population (both in terms of style and technique), knowledgeabout the skill acquisition process in terms of pattern formation,associated with movement architecture, etc., it is possible to modellong-term skill acquisition. This information can then be use to designlong-term training protocols that can account for broad individualcharacteristics such as morphological and physical characteristics. Theunderstanding of the development of the movement architecture can beapplied to refine an individual's motion structure. Capturing thedevelopment of the movement architecture through a formal hierarchicalmodel structure makes it possible to design training schedules thatallow systematic skill development paths based on the understanding ofthe refinement in movement architecture.

Capturing movement structure and skills provides a formal basis to tracka user's stage in the acquisition process. This understanding can thenin turn be used to determine the best forms of interventions, e.g. inthe form of feedback augmentation, at each stage along a skilldevelopment path. A key characteristic of a successful trainingframework is the ability to continuously deliver value to a user.Statistics provide most of the information after using the system for afew sessions, but then stop delivering new information. Subtle changesin movement performance are not captured by basic statistics. Forexample, the four typical outcomes in tennis (impact distribution,racket speed, spin and ball speed) do not provide sufficient directknowledge of the effect of training, fatigue, injury or even changes inthe equipment. Another reason is simply lack of measurements. Forexample, the role of footwork in the tennis stroke, or the hip motionand core, can only be captured indirectly by measuring the racketmotion.

It is possible to access more comprehensive characteristics of thebehavior by tracking different aspects of behavior in addition tooutcomes. However, extracting knowledge about these still relies onexpert interpretation. Tracking these characteristics over time canprovide valuable insights such as the onset of fatigue and the effect ofspecific injuries. These various factors may not sufficiently be pickedup in the outcomes since the body can compensate for them. However,tracking movement structure enables monitoring of specific changes inthe motion patterns. A persistent problem such as an injury willsufficiently alter the patterns, and thus be quickly identified. It willalso be possible to capture subtle changes that occur over time such aseffects of feedback augmentation or injury. With this system, it ispossible to give comprehensive diagnostics about the health of the userand to continuously deliver information that can be used to providedetailed adjustments to training or other aspects.

As will be appreciated by those skilled in the art, the physicalimplementation for a typical usage scenario of this system can berealized in a variety of systems and platforms. Following FIG. 1 andFIG. 6 , the general usage scenario is to track and assess skilliteratively while providing sets of movement instructions andaugmentation cues that enable the user's systematic development ofmovement skills.

This scenario is valid for many movement activities, and is consideredhere to contextualize the implementation of the platform. In thisscenario, the system components and computational processes areimplemented in the form of an application that runs on physicalcomponents found in the internet-of-things-type (IoT) platform,including: an embedded device, mobile host device, and cloudprocessor/database. These physical components implement the platform'ssystem components: sensor, host device, User Interface (UI), processing,storage, and augmentation. Delineating between the application, thephysical components, and the system components enables flexibledeployment across different activity domains and devices.

Most physical activities take place over a defined period of timedenoted ‘session’, possessing a training structure, specified mode(s) ofoperation, and user interactions. These sessions can include play, game,or training sessions.

A typical session is segmented into periods when the user is playing:‘sets’ or ‘runs’. Sets are delineated by intermissions in activity, whenthe user pauses to review activity or simply rest. Therefore, a typicalsession organization consists of cycles of sets and intermissions.

During each set, the sensor device captures activity measurements,preprocesses the measurements, and streams the measurement data or asubset of the data to a host device. The host device then furtherprocesses the data. Alternatively, the device can stream the datadirectly to the cloud, possibly relayed via the host device. Differentnetwork topologies and architectures can be conceived.

For example, a single player can engage with a mobile enabled devicethat communicates with the internet. Alternatively, two or more playerscan interact with one or more mobile enabled devices that communicatewith the internet. Each player can have an associated sensor device, oreach player can have equipment (e.g., tennis racket) with an embeddedsensor device.

As a sequence of sets unfolds, data can either be continuouslyprocessed, batch processed, or processed in a hybrid manner of the two.Processing allows activity review or summarization during anintermission, communicating relevant information to the just-completedactivity or future activity. Data can also be available continuously forreview, possibly by a trainer.

In one implementation, the user interacts with the system via hands-offfunctionality, such as a natural language interface. For example, inaddition to the configuring cueing functions, the player can issue aspecific request about training status or current or past performance.Specific training profiles can be selected, based on current skill. Atraining profile will activate a cueing profile, providing cue signalstailored to the specific goal of that training profile.

The typical user scenario considers a user who regularly performs andtrains skills for an activity over a period of months or years, and whowants to track progress and receive instructions for developing,maintaining or rehabilitating movement skill from session to session.More specifically, a user may perform an activity two to three times aweek, or even more frequently. The platform aggregates and assesses theuser's movement performance data within the assessment loop (see FIG. 1). This process generates a movement repertoire map and skill modelwhich describes the user's current and historical skill states.

At any time during the activity, the user can choose to activatefeedback augmentation, in the form of instructions and/or feedbackcueing (FIG. 12 ). Even if the user does not enable augmentations, theassessment loop can operate in a passive mode, continuing to collectdata whenever the user is engaged in an activity and continuallyupdating the skill model.

In general, the platform's system components can comprise of thefollowing: sensor, host device, User Interface (UI), processing,storage, and augmentation. The sensor is one or more components thatacquires movement data.

The host ensures that all of the system components are connected andtransfers data between them, in either afferent or efferent directions.The host device is typically not involved in data processing or enablingthe user interface.

The user interface (UI) is the component responsible for userinteractions with the platform: recording their interactions,translating the interaction into meaningful processing requests (e.g.,start recording, display skill metric, etc.), and delivering theresults. The UI does not necessarily have to be graphical in nature,operate completely in real-time, or even implement all of the previouslydescribed platform user interactions.

The platform's processing component is responsible for processing datafrom the sensor into relevant information, including, but not limitedto, parsed movement data, movement models, skill models, and training orcueing instructions, as detailed elsewhere. In addition to generatingrelevant information, the processing component prepares data for thestorage component, by organizing processed information into a data modeland data structure designed for hierarchical and contextual views ofmovement technique and activity performances. This entire process mayconsist of various levels of computations; some of these take place inreal-time.

At the highest level, the user interface is responsible for executingall end-user requests in processing requests, data requests, systemconfiguration changes, system activations, and many other tasks. Morespecifically, the user interface (UI) enables users to browse historiesof their movement performances, explore skill diagnostics, and engagethe assessment and augmentation loops of the training system. The UIallows the end-user to participate in the assessment loop by displayingcurrent skill status, diagnostics, instructions, and repertoire maps.When enabling the augmentation loop, the UI then provides options toselect from a variety of cueing profiles enabling and configuring theaugmentation component.

Without a coherent data structure, navigation of large amounts ofmovement performance data is intractable. The design of the UI componenttherefore is dictated by the skill and movement models for an activity.By ensuring the UI is designed with a specific skill model in mind, themodel can most effectively be leveraged to support the user'sunderstanding of their movement technique. The classes and phases thatdescribe a user's skill and motion models can themselves define elementsfor efficient communication of information and navigation on displaysystems (e.g., on a smart-phone, tablet, computer screen, etc.).

Specific quantities extracted from the skill model are used to producevisualizations and define an interface, as well as organize the userinteractions. These quantities include outputs such as temporalcontextual information, ensemble statistics, ensemble motion and phaseprofiles, and ensemble special motion and phase profiles. These outputscan be mapped to a visualization which includes a navigation interface,a repertoire map, and a phase view, all presented on a user device'sdisplay.

The aggregation of the data and its corresponding data structureorganization makes it possible to track the development of anindividual's movement patterns or changes to the patterns. The long-termaggregation describes the primary movement characteristics of anindividual and the short-term aggregation makes it possible tounderstand the efficacy and impact of a training intervention, cueinglaw, or other factor such as equipment or physical status, etc. Thislevel of information contributes to the definition of the repertoire maplevel of the interface.

The motion model's phase segmentation makes it possible to analyze anddisplay relevant quantities for each stroke phase. The phaserepresentation allows concise descriptions focusing on characteristicsspecific and relevant to that phase. In addition, building on inherentmovement architecture, the motion data inherits the hierarchicalstructure that also provides logical means of navigating and visualizingthe data. All three characteristics can be leveraged in the userinterface design, determining what information to display as well aswhat user interactions the system can elicit, at this level. Thistransparent and logical representation is essential for teaching andbroadcasting. Augmentation modality selection can be accomplished via agraphical 10 and/or natural language processing UI. Duringintermissions, or when there is sufficient time to review performance, asmart phone or tablet can provide convenient Graphical User Interface(GUI) interactions. During play, hands-off user inputs and feedback aremore practical. For example, in addition to activating cueing functions(e.g., by speaking commands such as “THE SYSTEM: cue me on backhand topspin medium”), the player can receive speech prompts of the trainingstatus, as well as make inquiries about movement technique or outcomes(e.g., by speaking commands such as “THE SYSTEM: what was top spin onlast stroke?”).

Consider the tennis example use case. The tree data structure shown inFIG. 11 allows users to navigate through the different stroke phasesfrom a stroke 1110, for different categories, classes and types wherethe stroke 1110 can be classified into a category 1120 (e.g.,groundstroke, overhead, volley, serve), a type 1130 (forehand,backhand), and a class 1140 (FH Class A, FH Class B, FH Class X).Additionally, the stroke 1110 can be segmented into a phase 1150(impact, follow through, recovery, backswing, back loop, forward swing)which has attributes 1160. These plots can be used to highlight thecorresponding trajectory segment along with the performance and skillattributes that are derived from the characteristic phase features.

FIG. 19A shows how this tree structure can be translated into agraphical UI. The user can go deeper to bring up a “Phase View” (FIG.19B).

In order to enable long-term analysis and feedback on movementperformances and skill, historical records are stored. This storageconsists of many different entities, objects, and data structures withinthe platform:

-   -   (1) storage of raw motion data and processed motion data into a        hierarchical and contextual data structure;    -   (2) a user's motion model; and    -   (3) a user's skill model. Information can be stored on any        physical component, as long as the information can be accessed        by the other relevant components in the system.

A partial basis for the data structure is provided by the hierarchicalmotion and skill models derived from the inherent movement architecture.As will be appreciated by those skilled in the art, other activitiesshare similar general structure making the platform generalizable toother activity domains.

Changes in proficiency or skill can be assessed via repertoire map,which is part of an ensemble mode. To fill the repertoire map, data mustbe collected over longer periods (e.g. several months), making itpossible to capture the variety of patterns that reflect individuals'technique and skills. Furthermore, movement patterns can be categorizedas ‘stable’, ‘unstable’ or ‘outliers’. For example, from time to timeevery individual can execute both near perfect as well as catastrophicmovements. Once sufficient data has been collected it is possible todifferentiate between these characteristics as it relates to an overallgeneral performance or training trajectory.

Different database queries are required to support the different displaymodes used to investigate performance patterns. These queries include,for example: current set in session; comparing performance ofconsecutive sets in same session; overall performance during session;comparison of two or more sessions in recent activity history (e.g.,when training the same skill in a previous session); overall performancein sessions across specific calendar periods; long-term trends. Thesequeries can be performed across the entire movement repertoire, orfocusing on specific categories, types, and/or classes of movements;they can also be focused on certain time ranges or use other selectioncriteria.

The movement database can also be queried to extract movements withspecific characteristics from the repertoire. These movements can thenbe used as objects to define the user interface. Typical objectsinclude, for example: “optimal” profiles that describe the best strokein the user's repertoire; or “catastrophic” profiles that can causeinjury. The best stroke is determined based on outcome coefficients andtrajectory statistics and can be studied over the entire movementprofile or within each phase. Information returned in this case mightinclude movement profile statistics, such as mean and variance acrossrelevant state profiles for a given movement type and phase. Otherstatistics include phase state profile envelopes. This can be used togenerate a comparison between stroke instances in a current training setand the optimal strokes from that set, session, or historical results.Movement outliers (“catastrophic” profiles) that describe faultymovement patterns can be used to generate profiles to avoid andassociated feedback.

The storage component also enables inter-player comparison of movementskill and technique, via comparing motion models, skill models, orthrough other criteria. Additionally, reference movements can be used,allowing the incorporation of best practices derived from a particulardomain's movement mechanics.

A user's motion model represents the specific details of how a user'sdata is taken in, processed, and movement units and phase segments areextracted. This model includes information such as finite state models,thresholds, or other parameters and definitions. Depending on theactivity, there can be a general template from which each user's motionmodel is derived from, or a user's motion model can be completelyderived from recorded data.

A user's skill model represents their current relationship betweenmovement techniques and outcomes. It also includes information such aswhich improvements should be prioritized or what has been historicallydifficult to change. As with a user's motion model, a skill model willhave varying degrees of user/activity specificity.

FIG. 11 illustrates a data tree structure associated with the tennisstroke. Other domains tend to have similar general organization ofclasses and phases of motion.

For tennis, the temporal organization can be described as follows. Asession starts when a player walks on the court and ends when the playerleaves the court.

A set is a continuous period of training or play interval. A set endswhen the player takes a break, perhaps, during a game or during trainingto rest or speak to a trainer.

A rally is a sequence of consecutive exchanges, typically ending when aplayer misses the ball or makes an unforced error. Within a rally,strokes are sequenced.

A stroke (or swing) is the interval encompassing a time period beforethe ball impact (stroke preparation), the impact time period, and a timeperiod past the recovery. The period between stroke preparation andfollowing its completion is usually called the ‘ready state’. The strokeis described by ‘stroke profiles’ which represent the measurement timehistories. A phase is a segment of the stroke which reflects relevantkinematics, dynamics, or human factors (motor control, perception,decision making) that manifest in the stroke organization. Table 2describes some of the associated information for tennis.

TABLE 2 Overview of the session data structure. Quality Type FieldsSession Contextual session and Date/time, location, player informationsurface, player, opponent, etc. Set Time indices Time in ‘session’identify the segments when players are in play. Rally Time indices in‘set’ Start and end of rallies. In game, also include score and serverid.

The highest-level processing is the synthesis of motion models, skillmodels and cueing laws (see FIG. 10 ). These processes are part of thealgorithm and model development. A function of these models is toprovide appropriate movement/skill abstractions, and the continuedevolution of these abstractions, as the individual's technique and skillevolves. With sufficient user data from a broad range of applied skillsand techniques, a library of movement models can be assembled. At thispoint, relevant models for a new user can be selected by evaluating theperformance of various models from the library. The models will alsoneed to be switched or adapted as a user's skills evolve.

Feedback synthesis refers to the part of processing that generatesinstructions to improve the outcomes of movement techniques based on acomputational assessment of a user's skill status (FIG. 9 box II andIII). The instructions generated by feedback synthesis are(computationally) represented as differentials in the movement modelwhich correspond to variations outcomes. From an implementationperspective, the skill model, diagnostic functions, and the movementmodel and associated phase segmentation can all be used in generatinginstructions. More specifically, the feedback synthesis process includesthe following steps:

-   -   (1) Determine the movement features (feature outcome        derivatives) from the movement model of a user that the user has        the ability to change based on the potential or observed        constraints of an instruction (range of motion, ability to        accelerate movement, etc.); i.e., determine which movement        parameters are free to be optimized.    -   (2) Determine movement features that display a significant        impact on an outcome; i.e., select movement parameters that        should be optimized.    -   (3) Prioritize movement features that are both changeable and        have the most significant impact on outcomes; i.e., chose the        optimization step.

This analysis becomes the basis for personalized and individualizedinstructions, used to improve the movement technique and outcomes of auser's movement performance. The output of the feedback synthesis can betransformed into a range of feedbacks including: symbolic language(text, spoken), cues (visual, audible, haptic) or signals (visualstimuli, audible, haptic). The output of the feedback synthesis will betransformed into augmentations by the augmentation system component ofthe platform.

Computing is generally distributed across the platform, however mostcomputing functions related to movement and skill processing aretypically executed in the cloud. These processes require the mostcomputational resources and the most stored data to process, making thecloud a logical location. In some configurations, limited movement andskill processing can be executed at the device and host level. From animplementation perspective, the relationship between the data structure,storage component, and processing component is critical to effectivelyenabling the vast array of movement skill processing capabilities of thesystem.

The skill model assumes that the user learns patterns from a generaltemplate. The goal therefore is to identify the movement patterns thatbelong to the distinct schemas that are being learned. The first step isto parse movement patterns that correspond to the learned movementschemas or motor programs. The parsed data can then be aggregated toform the movement repertoire.

Movement classification takes the ensemble data from the movementrepertoire and determines a partitioning of this set into subsets ofsimilar patterns. The classification process is based on features thatare extracted from the time histories in the ensemble data. Thesefeatures are then classified using clustering algorithms such asK-Means. The motion profiles associated with the movement patterns canbe segmented into phases. Phases are identified from characteristicsfeatures which provide the basis for the detection rules that can beused for automatic segmentation. Different classes may display differentphases and phase segmentation rules.

Table 3 summarizes some of the relationships between the primaryprocesses, storage operations, and data structure attributes.

TABLE 3 Overview of the primary processes and database operations.Processes Data Structure Storage 1. n/a Temporal (session/set/etc.)Contextual Information 2. Time histories stroke Movement Profiles DataStreams parsing/ segmentation 3. Stroke profile aggregations Repertoire,outcomes, Relational information, and statistics associated statistics.statistical information 4. Stroke classification Category, type, classStroke parameters 5. Stroke phase segmentation Phase segmentation rulesUpdate stroke data 6. Phase analytics Technique/skill attributes Phasereference envelopes 7. Skill modeling and Skill Repertoire map, MovementParameterized user skill assessment class patterns, Outcome statusquantities 8. Motion model synthesis Motion Model Motion model parameter9. Augmentation synthesis Instructions, Cueing Hierarchical mapsrelating Profiles, Cueing Laws prioritization of change, methods ofchanges

The augmentation system component is responsible for delivering usefulinformation to a user in order to assist the development of the user'smovement skills. Augmentation can be packaged in forms such as graphics,text, speech instructions, audible cues, or haptics. Augmentation can bedelivered in real time, near real time, or non-real time and iscommunicated in two primary forms: instructions, which operate at theknowledge level, and feedback cues (see FIG. 6 ).

The augmentation component of the physical platform implements thefeedback modalities synthesized based on the skill diagnostics. Specifictypes of augmentation, as previously discussed, include: symboliclanguage (spoken, text), cues (visual, audible, haptic) or signals(visual stimuli, audible, haptic). The purpose of the component is toprovide feedback in appropriate forms to help a user meet a desiredperformance outcome.

FIG. 13 illustrates the organization of the primary functionalitiesacross the platform. Following an IoT type realization of the platform,the hierarchical organization can be divided into three parts: the cloudservers (database, processors), the local processors at a host level(user interface, host), and the device or user level augmented interface(augmentation, sensors).

The architecture shown in FIG. 6 also provides a blueprint for theinfrastructure. Some configurations will have processes that can bedirected toward the cloud (afferent path) and some other processes thatcan be directed toward the user (efferent path).

Additional components and sub-components can be included. The embeddeddevice can have a device manager (with device configuration, which iscapable of measurement and preprocessing, data parsing, featureextraction and data streaming), a sampling and pre-processing component(for calibration and filtering sensor characteristics, and sampling orresampling of application requirements), and a motion estimator (whichcoordinates and transforms data).

The mobile host has a mobile app which allows control over interactionswith the platform such as training profile selection, diagnostics ofperformance, and providing performance review. Additionally, the mobileapp can assist with movement parsing (e.g., profile feature extraction,impact analysis, stroke profiles and models, classification,aggregation) in some circumstances.

The cloud server performs stroke classification, stroke phasesegmentation, and technique or skill analysis. Additionally, movementdata processing such as phase segmentation, classification, featureextraction, and aggregation are performed by the cloud server. The cloudserver also provides activity analysis, motion models, skill analytics,and cueing laws. Statistical information can also be generated in thecloud.

As will be appreciated by those skilled in the art, the assignment ofthe system components to the three physical components is primarily madeto explain the distributed nature of the platform, and differenttopologies can be used depending on the particular activity andscenario.

The host device is a computing device, of which possible forms include asmartphone, tablet, smartwatch, etc. In general, the host device can beconfigurable implement the following system components: host, UI,processing, and augmentation. In some configurations, the host devicecan implement the processing system component. However, since storageand computing power is typically limited on host devices, processingcapabilities on a host could be limited. Additionally, depending on thelatency between the device and the host acquiring movement data, somereal time feedback can be provided.

Host device functionalities can include data routing, request routing,and providing a control layer between the system components. Forexample, the host device can be responsible for getting data from thesensor to the processing and storage components, as well as returningresults from those two components, routing UI requests, and potentiallyhandling feedback configuration. This means that the host possessesknowledge of the sensor and the sensor's data format, as well asknowledge of what data form the storage and processing componentsexpect.

The host can manage any required data translation/transformation betweenthese other components. For systems with limited performance wherevarious forms of preprocessing are in use (e.g., compression), the hosthandles some of this work. The host can also trigger system processessuch as: when processing is performed, when data is transferred totemporary or permanent storage, when augmentation configurations areupdated, and other system control decisions.

The cloud physical component is a database and application server toimplement the storage and processing system components, for a singleuser and/or a population of users. The cloud archives and organizesmovement data, making various data analysis and data management servicespossible. For example, in the typical IoT deployment of the platform,the stored and processed data can be used to extract relevantinformation about the user's performance. Centralizing data in the cloudalso makes it possible to perform machine learning functions that can beused to determine motion and skill models, cueing laws, etc. and toinvestigate how these models may fare across a user population. As willbe appreciated by those skilled in the art, the function of the cloudcomponent can change during implementation. For example, generation ofUI elements can be performed by the cloud component, and some functionsattributed to the cloud can instead be implemented on the host or eventhe device.

The device can be an embedded computer that is able to remotely or viaphysical attachment observe or record a piece of equipment, the user'sbody (or portions thereof), or environmental factors. The device canimplement the sensor and/or augmentation system components. For manyuses, a likely implementation of the device would combine motion sensors(such as an inertial measurement unit, IMU), a processing unit, a memoryunit, a wireless radio for transmission and reception of data, and atransducer for generating feedback.

Relevant measurements to capture include a user's movements, themovement of equipment, or the movement of an end effector manipulated bya user. The device can either be attached or embedded in the equipmentand/or worn by the user. The sensor captures movement units relevant foranalysis and to support augmentations. In addition to motion data, othermeasurements include surface electromyography (EMG), forces, etc.

The data format will depend on the specific sensors. For IMU basedsystems, this is commonly time series data of motions. For computervision systems, relevant quantities can include raw video files,extracted stills, or time series data of extracted skeletal bodies. Datacan be either be continuous over entire sessions, limited to regions ofinterest, or consist of already extracted movement units.

For any data captured, there are corresponding sensor configurationparameters. These parameters include scaling factors, transformations,active sensors, sample rates, measurement units, and other measurementsystem parameters. In order for data to be processed effectively,knowledge of these parameters is required. However, these parameters canbe communicated in a separate piece of information (metadata), embeddedwithin the sensor data, already applied to the data, hard-coded on allends, or recorded in some other fashion.

The device can also be responsible, in part, for supporting theaugmentation system component. This includes an implementation of thecue processor and cue generator, as described in FIG. 10 . Such a devicealso requires some form of transducer capable of generating an audio,visual or haptic signal or stimulus. The type of transducer isdetermined based on what stimuli can be best recognized by the userwhile engaged in a particular activity. The cue generator can beintegrated with the device or can be part of other accessories ordevices such as a smartwatch, eyeglasses, or earpiece. A cue generatorcan be connectable to the platform components (in particular the deviceand host) via a wired or wireless connection.

A device including at least one such sensor is shown in FIGS. 23A and23B. Such a device combines inertial measurement unit 2350 (IMU) andother motion sensor device (e.g., GPS, magnetometer, etc.) with a CPU, aradio 2340 for the communication to a host or internet. The sensorcomponent does not have to be a separate device, and can be integratedinto the host physical component. Additional components include, forexample, an on/off switch 2360, a voltage regulator 2320, a clock 2330,a speaker 2370 (or buzzer), or a USB port 2310.

A variety of configurations of the system components and physicalcomponents and devices, based on the commercially available technologiesof the IoT ecosystem, can be employed without departing from the scopeof the disclosure. The IoT ecosystem combines host, computing resources,sensors, mobile and internetworks, as well as other components such assmartwatches, (near) local distributed computing processors, replicatedstorage systems, and more. The precise location used for theimplementation of the system components of this platform is determinedby practical considerations specific to the activity. The assumption forthese configurations is that a user will be either performing ortraining movement skills. Augmentations and feedback is typicallyscheduled and deployed through a training agent application thatembodies the UI, processing, and augmentation system components. Thesensor component in these configurations enables measurement andtracking of movements while the UI component gives a user access totraining schedules and training sets generated from the skilldiagnostics. The training agent application is typically deployed andaccessed by the user on the same device as the UI (for example, a mobilephone or smartwatch).

In one implementation of the deployment of the platform systemcomponents, the user can activate the application running on asmartwatch to receive a set of instructions, then activate real timescues to guide the performance of a specific movement type during thelive performance of the movement. In this usage mode, before the userstarts the current or next activity set, or during an intermission fromthe activity, the user can look at their device display where theapplication shows a specific set of technique instructions calculated toimprove outcomes for the movement type (see FIG. 6 ).

One configuration has a piece of “smart equipment”. The smart equipmentprovides real time cueing to guide a user's movement technique viameasuring the equipment's motion. The piece of equipment (e.g., racket,golf club) contains the physical device, implementing the sensor andaugmentation system components. A smartphone implements host and UIsystem components. The cloud implements processing and storage systemcomponents. These three components are in communication through wirelessradio technology. For example, the smart equipment communicates with thehost via Bluetooth Low Energy (BLE), and the host communicates to thecloud via Wi-Fi or a cellular network.

Another configuration is a stand-alone device with real-time cueing,where the stand-alone device measures the activity movement performanceand delivers real time cueing. This essentially separates the device andequipment physical components described in the previous configuration.The device physical component can then exist in various form factorsthat are appropriate to the requirements of the activity, either worn bythe user or attached to equipment. The other system components can bethe same as the previous configuration example.

Another configuration is a smartwatch as a real-time cueing device. Thesmartwatch implements many of the major platform functions. It can beworn by a user during the activity and can provide feedback. Thesmartwatch serves as the device and host physical components,implementing the sensor, UI, host, and augmentation system components.In such an embodiment, some of the processing system componentactivities are implemented on the smartwatch based on available systemresources such as storage and processing power. The cloud physicalcomponent implements processing and storage system components.

An alternate configuration is a stand-alone device as a sensor, with asmartwatch operating as a host and providing augmentation; the deviceeither being worn or embedded into “smart equipment”. This configurationallows for the precision of sensors embedded directly into equipment ormeasuring specific body activities, while providing the convenience of awearable user interface, display, and audio capabilities available incommercial smartwatches. In such an embodiment, some of the processingsystem component activities are implemented on the smartwatch based onavailable system resources such as storage and processing power. Thecloud physical component implements processing and storage systemcomponents.

EXAMPLES

A. Tennis

The tennis processing follows the organization provided in FIG. 9 andthe overall operation is divided between the assessment and theaugmentation loops. The data processing for the assessment loop is doneon an as-needed basis. Once the motion data has been aggregated,processed, organized into distinct classes and segmented into phases,the information is incorporated in the motion model 950. This model isused to assess and diagnose the individual's technique and skills andultimately synthesize the instructions and feedback augmentations 930.The former are delivered to the user using a communication system (see621 in FIG. 6 ) via a host such as a smart phone or smartwatch. Thelatter are delivered to the user via the cueing system 624.

The example is used to demonstrate how the processes and analysistechniques using motion data collected from the IMU device 1830 attachedto a tennis racket (see FIG. 18A).

For the tennis example, the motion sensors are attached to the racket.The purpose of the sensors is to capture the racquet's motion over thestroke cycle. There are no explicit measurements of the arms or otherbody segments. In another implementation, additional sensor units,placed at various locations such as feet or arm, can be used to capturethe more of the complete movement system behavior, enabling analysis ofhow the other body segments participate in the stroke movement andpotentially identifying other elements relevant to movement technique,such as posture or footwork.

FIG. 18A shows the coordinate system for the IMU sensor unit 1830{x_(s); y_(s); z_(s)} and the tennis racket frame body coordinate system{x_(r); y_(r); z_(r)}. FIG. 18B illustrates the impact datum frame for aright-handed user's forehand.

Note that the orientations of the frames are arbitrary; equivalence toother orientations can be shown. The racket body frame (RBF) coordinatesystem is shown at the racket's center of gravity. The position vectorr_(s) defines the origin of the IMU's coordinate system relative to theorigin of the racket's coordinate system.

The accelerometer measures the racket's acceleration along the directionof the x_(s), y_(s) and z_(s) axes and are denoted a_(x); a_(y); a_(z).The angular rate sensor measures the racket's angular rate about thex_(s), y_(s) and z_(s) axes and are denoted ω_(x); ω_(y); ω_(z). The IMUdata is filtered and transformed from the sensor frame to the RBF.Additionally, as will be appreciated by those skilled in the art, thesensor's accelerometer and angular rate sensor axes may or may not be onthe same device or component (there may be two IMUS).

At the most abstracted level of tennis skill analysis, behavioral eventsare primarily related to game structure and rules that dictate scoring.This level includes events such as shot selection, and patterns of playrelating to the game plan. At the intermediate level, the behaviorsconsist of body motion relating to court positioning and decisionsfocused on stroke selection and ball placement. These behaviors areprimarily determined by the game structure and are considered to bedecoupled from the stroke movement.

Below these levels are the core behaviors pertaining to stroke motionskill, namely the organization of the stroke execution in relationshipto the ball trajectory and opponent's movement. Further below, behaviorsconsist of the postural control subserving the stroke, dealing withintrinsic structure of movement implementation and organization and aremostly unconscious. At the stroke level, movement phases unfolddynamically. The transition between the phases is dictated in part byfunctional elements. For example, the ‘forward swing’ phase initiationhas to be timed based on the estimated impact conditions. Thisinformation is determined from perceptual cues extracted from the balltrajectory.

Following an optimal motor programing formulation, specifying the rackettrajectory and entire human body movement coordination to accommodatethe entire range of possible initial racket state, oncoming ballconditions, etc., a player would need to learn very extensive motorprograms. These plans would have to be learned for all the desiredoutcome conditions (spin, pace, etc.). This movement programing approachwould be intractable.

Humans exploit invariants in their sensory-motor behavior to mitigatecomplexity in the interaction with the environment. Therefore, whileevery stroke has unique characteristics due to the combination ofconfigurations (trajectory of the oncoming ball, its effect on the courtand player's positioning, posture, etc.), trained humans will learnefficient sensory-motor strategies to return the ball. Because of thecomplexity of the infinite range of conditions and the human body'sinherent constraints (sensory, perceptual, biomechanics, etc.), theworkspace that describes all interactions (sensory and motor behaviors)is discretized in sensory-motor primitives, each covering a specificrange of conditions and outcomes.

To analyze a human playing tennis, comprehensive analysis of the unit ofbehavior should encompass the complete interaction with the environment,including the perception of the environment and task elements (playercourt position, adversary position, ball position and velocity), as wellas the complete body movement (footwork, postural, etc.; see FIG. 2 ).Since the sensory-motor behavior is directed at the implementation ofthe primary unit of behavior in the activity, sufficient details can beextracted through the measurement and analysis of the racket motionalone; these details by themselves already provide useful augmentationand illustrate the platform's capabilities. More sensors could enableanalysis of the larger factors that contribute to players' skills over avariety of different activities (see FIGS. 4A-4E). For simplicity,without loss of generality, the following considers the stroke describedin terms of the racket trajectory as a primary unit of movementbehavior.

The stroke is used to control the ball trajectory on the court. Tennisstrokes can be described as cyclic or aperiodic, goal-directed motions.The trajectory of the racket (and possibly other user's body segments)over an entire cycle is called the stroke profile. The motion's primarygoal is the controlled impact of the racket and the ball. Therefore, thetennis stroke is a type of interception behavior. The movement phasesbefore the impact are directed at controlling the impact conditions. Thestroke phase after the impact is primarily for recovery of the rackettrajectory to a resting state. Between consecutive strokes, the periodicmotion may be interrupted for a short time, depending mostly on theball's pace and trajectory and the players' court positioning.

The tennis stroke motions are organized as a sequence of phases (forwardswing, impact, follow through, etc.). The movement phases are a commonpart of teaching and training tennis. However, references to thesephases are typically qualitative. Organized stroke segmentation providesan objective, deterministic description of the stroke architecture basedon racket motion measurement data. The following analysis first focuseson the description of the stroke characteristics based on the racket'sangular rate and acceleration obtained from the IMU's measurements. Aprimary reference point for building a phase segmentation is the impact.The general stroke architecture can be described based on characteristickinematic features.

The typical stages or phases of a ground stroke are: Backswing: firststage of a new stroke, the racket accelerates backwards. Back loop:transition between the backward motion of the backswing and the forwardmotion of the forward swing. During that phase the racket essentiallyreverses direction following a looping trajectory.

Forward swing: acceleration of the racket to gain momentum for theimpact. Impact: the time interval during which the ball is in contactwith the racket and string bed. Follow through: racket movementfollowing impact, mostly resulting from the racket momentum gainedduring the forward swing. Recovery: transition from the follow throughto the ready state, primarily a deceleration of the racket and a changeof orientation before the ready state. Ready state: the “holding state”between the completion of one stroke and the initiation of the nextstroke with the racket typically held somewhere between the front andthe dominant hand side.

The recorded time series data from the sensor is first parsed into theprimary movement units for the activity, which in this example are thetennis strokes. The segments are then aggregated into ensembles andclassified according to stroke category. The stroke profiles within eachclass are then segmented into movement phases that reflect that strokeclass's movement architecture. The segments are decomposed into musclesynergies that can be associated with the pattern of muscle activationsand biomechanical elements. These steps provide the basis for the datastructure used for the motion skill analysis.

The first step in the data processing illustrated in FIG. 9 is parsingthe stroke measurement 918 to extract segments that correspond to fullstroke profile periods. In the present example, the parsing of themotion data into stroke profiles is performed using features from theball impact event to detect the motion pattern activity. The impact isidentified by finding regions of data where the following evaluates astrue:

(∥α∥)>a _(th) _(imp,th) )∧(∥j∥>j _(th) _(imp,th) )∧(|{dot over (H)} _(z)|>{dot over (H)} _(z) _(imp,th) )  EQ. 10

Throughout the data stream, each time the above expression is foundtrue, an impact index is created. Starting with the first impactlocation, the index is recorded in a list of indices. Then the datastream is advanced by a time period (possibly a user/activity/eventspecific parameter), and the next location where the above expressionholds is found and recorded, and the data stream advances again; thisprocess repeats until the entire data stream has been processed.

Subsequently, the stroke profiles are extracted by capturing the ranges(t_(imp)−ΔT_(pre-imp), t_(imp) ΔT_(post-imp)) for each recorded index &time. Time intervals are used to define the initiation and terminationof the stroke profile, as the physical initiation and termination of thestroke can be more difficult to process. With this technique it isdifficult to identify initiations/terminations that are quasistationary, i.e., that take place at low-speed and low-acceleration.Other techniques that focus on other features can be used depending onthe measurements.

Table 4 provides an overview of the stroke data structure. Thisstructure includes quality (stroke, phases, actions), type (with timeindices), and fields (stroke category, phase, and actions).

TABLE 4 Overview of the stroke data structure Quality Type Fields StrokeTime indices in Stroke category (ground stroke, volley, serve, ‘rally’and stroke etc.), type (forehand, backhand), class (spin and typestrength). Phases Time indices in Phase including forward swing, impactzone, ‘stroke’ follow through, recovery, ready state, backswing, backloop. Actions Time indices in Actions including, intercept, impact, peakracket ‘phase’ response, etc.

Once extracted, these profiles can then be aggregated, classified andfurther analyzed. The tennis stroke classification is based on ahierarchical tree structure (FIG. 11 ). The stroke class structureincludes: ‘categories’={groundstroke, volley, serve}; ‘type’={forehand,backhand}; ‘spin class’={top spin, flat, slice}; and ‘strengthclass’={strong, medium, low}.

Different incoming ball trajectories, body movement and positioning, anddesired end effects on the return ball trajectories lead to a variety ofstroke types. After the measurement time histories are parsed theresulting movement profiles are classified 916 to create a user's map oftheir movement repertoire; more specifically for this tennis example, astroke map. This representation is the first level of analysis andsupports various functionalities such as data management andvisualization.

The classification method uses features extracted from stroke profileinclude measurements obtained from motion sensor, which is an IMU, andstate estimates including racket orientation. Features make it possibleto abstract the large amount of measurement data and therefore enablecomputationally efficient classification. The feature set extracted fromthe stroke profiles can then be used to perform clustering of thestrokes. Generally, only a subset of feature is needed to delineatebetween particular stroke types and the classification can also often beperformed hierarchically.

Once the stroke profiles have been organized into distinct classes,which share similar characteristics, it possible to proceed with moredetailed analysis of the strokes and their outcomes in each class. Acentral aspect of movement processing is the phase segmentation 914. Thedetailed information can then be used to support the analysis of aplayer's techniques and skills.

The strokes are motions that are directed deliberately by the player toproduce specific outcomes. The primary purpose of the stroke is tocontrol the ball trajectory, therefore outcomes can be assessed fromattributes of the impact and resulting ball trajectory. The outcomes canbe measured or estimated from available measurement data. In the presentexample, the outcomes are estimated from the racket body frameacceleration and angular rate data during the impact phase. The outcomesare described by the following variables: the spin imparted to the ball,the momentum transferred to the ball, as well as, the impact location onthe string bed.

Movement structure and organization is central for modeling andanalyzing skilled movement. The overall stroke profile depends onseveral factors including movement biomechanics, neuro-motor, cognitiveand perceptual. All these factors will manifest in the form andstructure of the stroke.

The simplest model of a tennis stroke consists of two phases: a racket'sforward motion directed at the ball impact and the backward motionrequired to “load” the forward motion. Depending on the skill level thestroke can be segmented into additional phases. The tennis literaturegenerally considers the following four phases: backswing, forward swing,impact and the follow through. In this example, the movementarchitecture (for ground strokes) comprises seven distinct phases. Inthe forward phase, the segments before and after the impact are the‘forward swing’ and the ‘follow through’, respectively. At both thefront and back ends, the stroke motion undergoes a change in direction.The end of the follow through is called the ‘recovery’ and the end ofthe backswing and transition to the forward swing is called here the‘back loop’. In addition, a ‘ready’ phase is considered to describe theresting period between strokes.

The phase segmentation provides a mathematical description of thesephases to allow for automatic detection and segmentation. Phases canusually be identified based on kinematic and dynamic characteristics.The phase segmentation uses features from the stroke profiles to detectphases in the stroke execution. Each movement phase is represented by adiscrete state in a finite-sate model which provides an abstract modelof the stroke motion. The finite-state model describes the overallstroke as the transition between a finite number of states. The motionmodel serves as basis for the motion skill analysis and real-timecueing.

After the stroke have been parsed, i.e. detected and extracted from themeasurement time series, they are aggregated in ensembles andclassified. In this example, the classification is performedhierarchically based on physical understanding of the major strokeclasses illustrated in the stroke tree (FIG. 11 ).

Information used to classify strokes can be obtained from a set offeatures extracted from stroke profiles (e.g., RBF angular rates andacceleration). These features capture characteristics of the strokephysics and allow to abstract the stroke and player's technique using aminimal set of parameters. Note that data about the outcomes could intheory also be used. In this example, the outcomes are only used for theskill analysis. The stroke features include:

-   -   (1) temporal indices such as those associated with the stroke        phases, including the swing rate zero-crossings (are related        with the initiation of the forward swing and the ending of the        follow through);    -   (2) state values at those temporal indices; and    -   (3) state values at various time samples before and after the        impact.

For example, the swing rate r at t=0.2, 0.1 and 0.02 sec before impact.

The stroke phase features are the features that are used to identify thestroke phases. These features may also provide information for strokeclassification. Table 5 summarizes some of the stroke phase featuresfrom the angular rate data.

TABLE S Stroke Phase Features t_(imp) impact time t_(fwd) beginning offorward stroke t_(rec) end of follow through, beginning of recoveryt_(p=0,0) roll rate zero crossing before impact t_(p=0,1) roll rate zerocrossing after impact t_(r=0,0) swing rate zero crossing before impactt_(r=0,1) swing rate zero crossing after impact

The stroke profile features are chosen to capture the shape of thestroke profiles. Their primary purpose is stroke classification. Theyessentially are a low-dimensional description of the profile shape. Thegoal is to use features that are maximally informative to differentiatebetween stroke types and execution or technique. The phase featuressample the RBF acceleration and angular rate measurements over a smallnumber of specific time samples across the stroke profile. These pointsare chosen to capture important profile characteristics such as theswing rate early in the forward swing and shortly before the impact.Such features can be identified from principal component analysis (PCA)of an ensemble of representative stroke profiles.

The forward swing is the primary motion toward the impact, thereforefeatures extracted from the stroke profiles for that phase arepredictive of the stroke type and quality. The stroke kinematic anddynamic characteristics just before impact (interception phase) provideinformation about the desired impact conditions. Similarly, the followthrough profile characteristics provide information about the stroketype and technique. Table 6 summarizes the stroke profile features. Theprofile features capture the stroke profile characteristics. Finally,external characteristics can also provide valuable information. Table 7describes peak and statistical characteristics of the stroke profile.

TABLE 6 Stroke Profile Features p_(imp-0.2s) roll rate at the beginningof interception. p_(imp-0.1s) roll rate early in forward swing.p_(imp-0.02s) roll rate just before impact. q_(imp-0.2s) pitch rate atthe beginning of interception. q_(imp-0.1s) pitch rate early in forwardswing. q_(imp-0.02s) pitch rate just before impact. r_(imp-0.2s) swingrate at the beginning of interception. r_(imp-0.ls) swing rate early inforward swing. r_(imp-0.02s) swing rate just before impact.r_(x,imp-0.2s) axial acc. at the beginning of interception.a_(x,imp-0.1s) axial acc. early in forward swing. a_(x,imp-0.02s) axialacc. just before impact. a_(y,imp-0.2s) transversal acc. at thebeginning of interception. a_(y,imp-0.1s) transversal acc. early inforward swing. a_(y,imp-0.02s) transversal acc. just before impact.a_(z,imp-0.2s) transversal acc. at the beginning of interception.a_(z,imp-0.1s) transversal acc. early in forward swing. a_(z,imp-0.02s)transversal acc. just before impact.

The stroke classes can be characterized hierarchically according tostroke categories (ground stroke, volley, serve), types(forehand/backhand) and classes (spin and strength). There exists adeterministic relationship between the racket motion, as described bythe RBF acceleration and angular rate components, and the basic strokecategories, types and classes. In the following example, knowledge ofthe stroke mechanics is used to determine physical criteria for strokeclassification and these criteria are used to determine the necessaryfeatures.

The hierarchical classification process based on physical featuresdiscussed next. Once the features have been extracted from the strokeprofiles the classification first delineates between stroke type(forehand/backhand), second based on the ‘spin’ class(topspin/flat/slice), third in terms of the stroke strength(high/med/low).

TABLE 7 Stroke Statistical Features {circumflex over (q)}₀ pitch rateextrema before impact {circumflex over (q)}₁ pitch rate extrema afterimpact σ_(q,fwd) pitch rate variance during forward swing σ_(q,fol)pitch rate variance during follow through t_(|ax|<10) time when thelongitudinal acceleration drops below 10m/s² after the impact â_(y0)transversal acc. extrema during forward swing. â_(y0) transversal acc.extrema during forward swing.

As an example of empirical classification, there is a relationshipbetween racket longitudinal and vertical accelerations a_(x) and a_(z),and swing rate r. Features can be extracted from the swing rate r andlongitudinal acceleration a_(x) at 20 msec before impact. Thedistribution of these features for forehand and backhand strokes at 20msec before impact illustrates how these features cluster in two groupsaccording to the forehand FH and backhand BH. The simple classificationrule (for z down) is given as:

FH: r(t=−0.02)<0  EQ. 11

BH: r(t=−0.02)>0  EQ. 12

Similar to the FH/BH classification, it is possible to definequantitative criteria for the stroke ‘spin’ class (topspin, flat andslice). The primary variables that determine the impact spin are theracket roll rate p (sometimes called “windshield wiper” motion)potentially with some contribution of the racket pitch rate q.Therefore, information about these quantities shortly before impact canbe used for stroke classification. The following data assumes that theRBF axes configuration with z_(r) down (see FIG. 18A. In the followingexample a spin class feature is defined based on the racket roll rate pand swing rate features r 20 msec before impact.

The swing rate distribution can be divided into the FH and BH. The rollrate breaks up in the positive p>0 and negative p<0 roll rate for thebackhand (r>0) and two clusters in the forehand. These clusters reflectthe amount of slice (or backspin) and top spin. There are also a strokeswith low roll rate which are characteristic of a low spin, i.e. flatstroke. A threshold for flat strokes _(P)flat can be used to delineatebetween flat, topspin and slice.

Topspin: p(t=−0.02)<_(P)flat  EQ. 13

Flat: |p(t=−0.02)|<_(P)flat  EQ. 14

Slice: p(t=−0.02)>_(P)flat  EQ. 15

The threshold value _(P)flat can be set by hand or can be determinedfrom clustering results associated with a player's technique.

Another useful classification is the strength of the stroke based onchange in racket momentum during the forward swing. Note that this isnot necessarily the same as the impact strength which is related to themagnitude of the racket momentum change at impact. The followingemphasizes is on stroke technique, therefore the strength of the strokebased on the stroke characteristics before the impact will be used here.Same ideas can be applied to the impact strength. Two strengththresholds of 50 and 100 m/sec² have been used to delineate between low,medium and high strength classes, which can be seen from the colorhighlighting.

High: |a _(tot)(t=−0.02)|>a _(med)  EQ. 16

Medium: a _(low) <|a _(tot)(t=−0.02)|<a _(med)  EQ. 17

Low: a _(tot)(t=−0.02)|<a _(low)  EQ. 18

These threshold values are set by hand. Similar to other strokecharacteristics these could also be set automatically based on resultsof data clustering.

The screenshot 1910 in FIG. 19A shows classification results for about1300 ground strokes that have been classified into 18 stroke classesfollowing the three criteria described in the preceding sectionsproceeding hierarchically. The strokes are first classified in terms ofthe type (forehand/backhand), next the spin class (top spin/flat/slice)and subsequently in terms of the strength (high/medium/low).

Once the basic movement patterns (here the tennis strokes) have beenparsed and classified, the next step is the decomposition of movementprofile data into segments that correspond to movement phases. Typicalground stroke architecture and the movement phases were describedearlier.

Phases are associated with the human motor control and biomechanics.These effects manifest in the movement kinematics and dynamiccharacteristics. Phase segmentation techniques are based onidentification of features that are characteristic of phase inparticular phase transitions. For example, particular changes in theacceleration and angular rates, such as zero crossings, that indicate achange in movement direction or changes in trajectory curvature.

The phase decomposition enables a more detailed description of thestroke that relates movement characteristics to underlying structuraland functional factors. Each phase can be described as discrete statesin a finite-state model. This model describes the movement within aphase as continuous dynamics (e.g. the F_(i) described earlier)transition between the phases based on state conditions that have beenidentified. The finite-state model is the basis for the formulation offinite-state machine used for finite-state estimation.

Observation from the rate and accelerometer measurements shows thatstroke profile ensemble contains distinct patterns that can be relatedto tennis stroke phases. These patterns are briefly discussed here. Moredetails about feature elements specific to the stroke phases.

FIG. 16A highlights angular rate features related to the phasedecomposition. FIGS. 16C-1 to 16C-4 show the phase portrait of theangular rate vector, which illustrates the 3DOF angular motionassociated with the stroke for four stroke classes (BH TS and BH SL, FHTS and FH SL). The angular rate plots highlight that the couplingbetween rate components is important in delineating the stroke phases.In fact, from the dynamics standpoint the complete angular phase-spaceis 6 dimensional (3 angular rates+3 angles). From a behavioralstandpoint, the stroke angular motion is probably a subspace with lessthan 6 dimensions. The fact that the sensed body rates describe awell-defined manifold in the 3D plots of FIGS. 16C-1 to 16C-4 indicatesthat the rates alone are probably sufficient to determine a large partof the stroke behavior.

The angular rate measurements provided by the IMU's gyroscopes areinvariant to changes in reference frame. The racket accelerationmeasurements, on the other hand, are user to the gravitation effect,which require knowledge of the racket's absolute orientation, as well aseffects of reference frame relative motion. The magnitude of theracket's inertial acceleration is much larger than the gravitationalacceleration, therefore, the racket acceleration may provide additionalfeatures for phase detection. The most notable feature is the racketacceleration during impact as well as before and following the impact.These are the stroke's most dynamic movement phases.

The analysis of the data in phase space, such as the description of thethree racket angular rates (p, q, r) provide additional insights andcriteria for the identification of stroke patterns. FIGS. 16C-1 to 16C-4show the 3-dimensional phase portrait for the angular rate andacceleration. These plots already incorporate results of the phasesegmentation.

The phase portraits describe the periodic patterns associated with thestroke. The plots show that the general shape has distinct andrepeatable elements. Even though the individual strokes have varyingcharacteristics there are several prominent features that underscore theconcept of motor equivalences associated in movement behavior. Asexpected the angular rate portraits are more consistent as theacceleration portraits. The trajectories are shown in terms of theindividual samples (sampling frequency of 1 kHz), therefore the samplespacing indicates the angular acceleration or jerk for the rate andacceleration portraits respectively. The segments where the sample arespread out correspond to the impact zone. The three-dimensional phaseportrait shows how the stroke follows distinct stages. Each stageevolves on a type of orbit in its own two-dimensional sub-space whichemphasize various coordination patterns between the angular degrees offreedom.

The phases described in the preceding sections define the primary statesof the finite-state stroke model (see FIG. 20A). Each phase ischaracterized by particular kinematic and dynamic characteristics. Thesecharacteristics can be used to formulate detection rules, which can beapplied to automatically segment the phases from the motion data. Thedetection rules also define the state transition in the finite-statemodel. Note that for many of these rules the measured quantities (hereRBF angular rates and acceleration) will give adequate phase transitioninformation only some of the time. Therefore, some phase transitionrules also include estimated quantities such as the orientation.

The transition rules between the stroke's finite states are as follows:Ready: the ready state or phase corresponds to the time interval betweenthe end of the recovery to the beginning to the initiation of the nextstroke, i.e. the beginning of the backswing until the next stroke. Theracket motion during the ready phase is quasi stationary, therefore theready phase period can be determined from the magnitude of theacceleration and angular rate measurements. Using a threshold on thetotal acceleration and angular rate:

t _(re) =t| _(∥ω∥<ω) _(re,th) _(∧∥a∥<a) _(re,th)   EQ. 19

An alternative approach is to limit the total post impact strokeduration. This rule can be more useful in situations where the racquetdoes not reach a quasi-stationary state, such as very fast paced games.

t _(re) =t _(0,imp) +ΔT _(dur)  EQ. 20

Backswing: the backswing corresponds to the first phase of a new stroke.As the name implies the state is initiated by the racket backswingmotion. The initiation of the backswing phase can be detected by usingthe time the azimuth rate reaches a given threshold and azimuth hasreached a threshold. Using azimuth (θ) is preferable to swing rate r asthe racquet grip style affects the swing rate.

t _(bs) =t| _(θ>θ) _(bs,th) _(∧{dot over (θ)}>{dot over (θ)}) _(ds,th)  EQ. 21

Back loop: the back loop is used to designate the transition phasebetween the backswing and the forward swing. This transition representsprimarily a rotational motion from the backward swing motion to theforward swing motion. This phase can be extracted from the time intervalinvolving relatively low acceleration but high angular rate, for exampleby evaluating the (quaternion unit sphere) trajectory curvature.

t _(bl) =t| _(κ>κbl,th)  EQ. 22

Forward swing: the forward swing is the phase that leads to the impact.Therefore, it requires sufficient racket head speed and precise contactconditions. It starts with the angular acceleration of the racket andends with the ball interception and eventually impact. The start of theforward swing is the time when the high-curvature trajectory of the backloop changes into a low-curvature trajectory as the racquet is set on aballistic path.

t _(bl) =t| _(κ<κ) _(bl,th)   EQ. 23

Note that it should also possible to detect the start of the forwardswing phase based on the total translational acceleration a_(tot). Thisrule will result in different phase delineation and different segments.

t _(0,swing) ^(a) =t| _(sgn(∥a∥(t−δt)−a) _(fs,th) _()≠sgn(∥a∥(t+δt)−a)_(fs,th) ₎  EQ. 24

Before the impact, there is a brief interception phase. Given theneuro-motor constraints (e.g. human neuromuscular constant and theresponse time) it is impossible to correct for the racket path as theracket closes in on the ball.

Impact: the impact onset is characterized by an impulsive accelerationassociated with the momentum transfer that occurs when the ball strikesthe string bed. As already described, the ball contact during impactlasts about 5 msec. The start of the impact is detected when the totalacceleration ascends through the threshold a_(th,imp):

$\begin{matrix}{t_{imp} = {t❘_{||a||{{> a_{th_{{imp},{th}}}} \land {{❘{❘j❘}❘} > j_{th_{{imp},{th}}}} \land {{❘{\overset{.}{H}}_{Z}❘} > {\overset{.}{H}}_{Z_{{imp},{th}}}}}}}} & {{EQ}.25}\end{matrix}$

The impact event is assumed to only last the period of time where theball and racquet are in contact. Although the impact can be shown to bea nonlinear event (and therefore not guaranteed to be invariant induration relative to impact strength), empirical study suggests thatmost impact events have a similar duration. Using duration is alsopreferable to examining rates/accelerations as it is not affected byracquet vibration modes.

t _(ft) =t _(0,imp) +ΔT _(imp)  EQ. 26

Follow through: the follow through primarily involves keeping control ofthe racket immediately following the impact and slowing the racket. Thismotion is mostly characterized by the limits on biomechanical range ofmotion. Similar to the forward swing, the follow through is primarilycharacterized by the change in racket angular deceleration. The end ofthe follow through can be detected by the racket swing rate r reversingsign.

t _(rec) =t| _(sgn(r(t−δt))≠sgn(r(t+δt)))  EQ. 27

Recovery: during the recovery the player takes the racket from the endof the follow through to a resting state which designates the readyposition.

The state-transition graph in FIG. 20A shows phases of the stroke whereeach phase defines a state. Each state is itself represented by itsspecific kinematics and dynamics and association with biomechanics andsensory, perceptual and motor control elements. These features andassociated structure provide the basis for the skill analysis as well asfor the design of player feedback mechanisms. The states of thefinite-state model are: ready, backswing, back loop, forward swing,impact, follow through, and recovery. Respectively, they are abbreviatedas: re, bs, bl, fs, imp, ft and rec.

FIG. 20A illustrates the finite-state stroke model with the seven phasesand examples of transition rules. An example of its operation is shownin FIG. 20B. The parameters in the phase transition rules are ΔT_(dur),θ_(bs,th), {dot over (θ)}_(bs,th), κ_(bl,th), a_(th) _(imp,th) , {dotover (J)}_(th) _(imp,th) , {dot over (H)}_(z) _(imp,th) , ΔT_(imp).These can be determined from statistical analysis.

In general, the stroke phases cannot be measured directly therefore theyneed to be estimated using a finite-state estimator. When limited to RBFaccelerations and angular rates determining the current state for arange of players with different styles and technique can be challenging.Therefore, the processing may not be applicable for these more generalcases. Furthermore, athletes might interrupt their motion in mid-stroke,leading to anomalous state transitions. A state estimator to determineracket orientation (attitude) can help make the finite state estimationmore robust. Finite-state estimation schemes based in statistical modelssuch as Hidden Markov Models (HMM) generally provide good accuracy androbustness.

The following describes the elements of the skill model and skillassessment for a tennis example. The three main components of the skillmodel illustrated in FIG. 9 , box III:

-   -   (a) Movement pattern repertoire, i.e. stroke map;    -   (b) Pattern phase profiles; and    -   (c) Movement synergies.

Table 8 illustrates the type of attributes that can be extracted fromthese data for technique and skill assessments.

First, concepts and metrics based on the repertoire. Understanding therelevance of the repertoire requires a brief description of the skillacquisition process focusing on the tennis use case. The typical tennisstroke repertoire comprises multiple classes of distinct swing patterns.These are acquired to achieve the range of outcomes needed to beproficient in a play and deal with the range of impact conditions. Thedevelopment of a repertoire of movement patterns is typical of openmotor skills described. The movements within each class form an ensembleof equivalent strokes that can be adapted to variations in impactconditions (speed, bounce height, spin, etc.) and outcome.

TABLE 8 Quantity Emphasis Attributes Session Play characteristics Totalplay duration, number of sessions, etc. Set Session characteristicsNumber of rallies, length of rallies, etc. Rally Exchangecharacteristics Number of exchanges in a rally, number of miss hits,etc. Stroke Stroke repertoire Number of stroke categories, types,characteristics classes (e.g. spin and strength), etc. Phases Strokeexecution and Phase profile characteristics (e.g. organizationvariability) and phase aggregate characteristics (e.g. duration, totalvariability) etc. Actions Specific functional aspects Stroke features(e.g. pitch rate range and statistics at 5 msec post impact, etc.)

The adaptation of the movement pattern within a given class, however,can only handle a finite amount of variations in impact conditions andoutcome. Beyond that admissible range, the stroke performance (e.g.energy or jerk) degrades and eventually reaches the limits of what isfeasible within a given movement pattern. These limits are typicallybiomechanical. To prevent the degradation or saturation, a new patternhas to be developed. Expert players have a broad repertoire of strokeprofiles. These patterns are also highly optimized and therefore anexpert player will seek to engage the ball using the optimal strokepattern and within the optimal impact conditions. Using optimal stroketechnique requires adequate stroke preparation, which depends on variousother facilities, including perceptual mechanisms, footwork and posturaladjustments.

A comprehensive stroke repertoire, therefore, gives the subject theability to use optimized, predictable stroke patterns instead of havingto rely on corrections during the stroke. The latter offer only limitedamounts of corrective or adaptive range given the various sensory motorconstraints such as reaction time and neuromuscular time constant.Beginners tend to gravitate towards using similar movement patterns andadapting those. Therefore, they exhibit poor pattern definition in theirrepertoire and movements in each class display large amounts ofvariability and the strokes are far from optimal.

In tennis, the movement pattern repertoire describes the stroke typesacquired by the user to accommodate to the range of oncoming balltrajectories and the range of effects and outcomes. Therefore, thestroke classes reflect both the range of conditions (pace, spin, heightof the oncoming ball), as well as, the range of desired stroke outcomes(ball pace, trajectory shape and length, etc.).

Note that the results that are presented in this example only focused onground strokes category. Note also that stroke maps are not limited tothese classes. Given sufficient data, collected under rich playconditions, data driven classification allows identifying broadercategories and additional types of classes. The repertoire is given interms of stroke categories and classes captures the variable andinvariable characteristics of someone's strokes.

Processing of the repertoire can be based on different dataaggregations. For example, different historical time periods can bemapped such as the past year, month, week, etc. The current session,set, or the past N strokes, etc. It is also possible to aggregate thestroke data based on attributes, such as the strokes for the N bestoutcomes. These capture different aspect of behavior includingidentifying deep rooted patterns and allows to track how trainingstrategies, etc. impact behavior at the repertoire level.

Finally, the repertoire assessment depends on the skill level. Anadvanced player requirement for a repertoire is different to that of abeginner. The advanced player's repertoire has a broader range ofpatterns and these patterns have a more developed movement structure(see FIG. 15E). Therefore, skill assessments have to be based on skilllevel. At the repertoire level, the expectations for the range ofpatterns and the performance of the patterns in terms of the outcomesare based on the movement structure and the task performance level.

In some applications the movement outcomes are not directly measured andtherefore have to be estimated. The following describes how the outcomecan be estimated from impact conditions described by racket IMUmeasurements during the impact phase. The spin and pace of the returningball are determined by the impact conditions and therefore sufficientmeasurements of the quantities involved in the impact can be sufficientto accurately estimate the outcomes. In another tennis application, theoutcome could be measured using cameras and computer vision algorithmsor even instrumenting the tennis ball with IMU sensors.

The ball is controlled by imparting linear and angular momentum from theracquet. Based on Newton's third law, the change in the racket's linearand angular momenta has to be equal to the change in the ball's linearand angular momenta (assuming the player does not impart any forceduring that phase). Since the collision is short in duration, the changein momenta can be attributed to the impact alone. To capture completespin and pace values for the tennis ball, a dedicated measurement systemsuch as a vision-based motion capture system is required. When workingwith limited sensors, such as the racket IMU 1830 (see FIG. 18A), theoutcomes can be estimated from the racket motion during the period ofimpact.

As a stroke includes many velocity/rate changes, there is never a periodwhere linear and angular momenta are constant. Therefore, isolatingmomentum transfer due to the ball impact, and not the player's input, ischallenging. One approach is to filter the data into low frequency andhigh frequency components. Subtracting the low frequency components fromthe original signal leaves (approximately) only the impact component ofthe signals.

With an estimate of the acceleration attributable to impact, a_(imp),and the angular velocity and acceleration attributable to impact,ω_(imp) and α_(imp), it is trivial to calculate momentum transfer, as itis the integral of force.

ΔL=∫ _(t) ₁ ^(t) ² ma _(imp) dt  EQ. 28

ΔH=∫ _(t) ₁ ^(t) ² M _(imp) dt  EQ. 29

where m is the mass of the racquet and M is the moment (torque) actingon the racquet, calculated from inertial measurements and knowledge ofthe racquet's inertia tensor I, defined as:

M _(imp) =Iα _(imp)+ω_(imp)×(I·ω _(imp))  EQ. 30

Since the mass and moment of inertia of the ball are significantlysmaller than the racket, the racket's state at the onset of the impactdominates the outcome, and the spin outcome can be estimated from theracquet's state alone. The impact location can be calculated from therelationship between translational and angular momentum transfer duringthe impact.

ΔH _(x) =m(ΔL _(z) r _(y) −ΔL _(y) r _(z))  EQ. 31

ΔH _(y) =mΔL _(x) r _(z)  EQ. 32

ΔH _(z) =mΔL _(x) r _(y)  EQ. 33

Where H, L and, m are as defined above, and r is the position vectorfrom the racquet mass center to the point of impact. Solving the aboveequations for the components of r gives the impact location. With theimpact location known, the ball's spin can be calculated. Assuming ano-slip impact condition (where the ball does not rotate with respect tothe racquet), the angular velocity of the ball can be calculated as thesum of the angular velocity of the racquet in the inertial frame and theangular velocity of the ball in the racquet frame; sum of the angularrate of the racquet and the out-of-racquet-plane velocity vector of thepoint of impact divided by the radius of the ball.

^(N) w ^(B)=^(N) w ^(R)+^(R) w ^(B)  EQ. 34

The angular velocity ^(R)w^(B) must then satisfy:

v ^(r,i)=^(R) w ^(B) ×r _(ball) {circumflex over (x)}  EQ. 35

where v^(r,i) is the velocity of the ball-racquet contact point, in theinertial frame.

For the analysis of tennis stroke skill, the goal is to determine theeffect of movement technique on outcome. This requires an analyticalframework to describe the relationship between the stroke technique andthe stroke outcome. Even if the outcome is determined by impactconditions, the racket state at impact depends on the entire movementcharacteristics leading to the impact with the ball.

In FIG. 16B, stroke profiles from a forehand stroke class are segmentedinto phases following movement organization and environment interactionsshown in FIG. 2 . The phase decomposition describes the particularstroke architecture acquired by a player. The stroke architecture isspecific to the outcome and conditions. The tennis stroke phases aresimilar to typical stroke notation in the literature, with thedifference that a segmentation based on movement dynamics provides aquantitative instead of qualitative results. The quantitative approachalso results in additional states such as the back loop phase. The phasestructure depends on the stroke pattern and the skills.

As visible from the angular rate phase portrait shown in FIGS. 16C-1 to16C-4 , different stroke profiles within the same stroke class areisomorphic. Each class can be recognized through its characteristicstate time-history profile (see the overall view of stroke profiles inthe stroke map in FIG. 19A).

Given oncoming ball condition (trajectory, height, pace, spin) anddesired outcome trajectory (determined by amount of top-spin, strengthand momentum transfer), calls for a specific initial racket state at theforward swing initiation FIG. 16D. The initial forward swing state, inturn defines the initial back loop state, etc. Slightly different muscleactivation patterns and phase timing are used to modulate the responsebut the overall stroke patterns preserve their general form. Theseinvariant properties are due to the fact that strokes in each classbelong to the same sensory-motor pattern.

The control of the ball trajectory is achieved by precisely controllingthe spin and pace of the ball leaving the racket. The tennis example,focuses on such as top spin or pace. The following describes the phaseprofiles characteristics for the skill assessment FIG. 9 Box III b).

The stroke starts about one second before the impact (see FIG. 16A) whenthe player initiates the backswing and extends beyond the impact. Thebackswing is triggered by cues used to anticipate the shot direction.These are extracted visually, e.g., from the opponent's movementbehavior, posture, etc. The stroke back loop and forward swing phasesare directed at the ball interception and the control of the impactconditions. As noted elsewhere, the forward swing is a consistent motionwith invariant conditions at about 200 msec before the impact time FIG.16A.

The variations within a class are due to compensation for changes inimpact conditions and modulation in outcome. FIG. 16B shows the racketorientation phase portrait for forehand strokes with different level ofspin outcomes. Variations, of course, are also produced by random motornoise associated with human performance. Variations that are notcorrelated with changes in the outcome are compensatory. In tennis forexample, adaptation is required to accommodate varying impact conditions(pace of the oncoming ball, impact height or spin) and modulatingoutcome (amount of top spin or pace). Variations that are correlatedwith the outcome highlight the functional properties of the movement.

The phase decomposition makes it is possible to determine relationshipsbetween specific phase segment features and outcomes. FIG. 16D shows anensemble of phase profile segments for the high-strength, forehand topspin (FHTS) and forehand slice class (FHSL) across the forward swing,impact and follow through phases. The plot highlights the phase profileenvelope in terms of the racket yaw rate r, the roll rate p and theelevation E.

These functional characteristics used to modulate outcome andcompensation for variations in impact condition can be modeled todetermine skill analysis and determine the mechanisms that can bemanipulated for the design of cueing or feedback mechanisms.

Movement skill in this tennis example can be defined as the ability ofplayers to effectively use their movement technique to achieve desiredoutcomes. Therefore, sensitivity analysis can be used to determine howlevels of outcome are related to movement technique. As was describedthis analysis makes it possible to determine specific characteristics ofan individual's movement technique that contribute to the outcome.

The movement profiles capture the movement performance and functionalcharacteristics. The state variables in each phase reflect howbiomechanics are used to attain the particular outcome and how thisstrategy is adapted to accommodate impact conditions. Many combinationsof variables can potentially influence the outcome, and determining eachrelationship exhaustively can be inefficient. The movement model makesit possible to extract features that explain the underlying controlmechanisms used for adjusting the level of outcome and adaptation toconditions.

The spin imparted to the ball is a function of combination of racket andball states at the impact, x_(r) and x_(b), respectively. The state atthe impact is a function of the racket trajectory x, which is given byits states including for example, the racket roll rate, pitch rate,heave rate, azimuth rate and elevation rate:

S=f(x _(r)(t _(imp)),x _(b)(t _(imp)))  EQ. 36

The racket impact conditions can only be controlled indirectly throughthe stroke; the ball's state cannot be controlled. The impact condition,and hence the outcome is a function of the racket trajectory:

x _(r)(t)=x _(r)(p(t),q(t),r(t),{dot over (A)}z(t),ĖL(t), . . . )  EQ.37

Given the model in EQ. 36. In theory, it is possible to determine howperturbations in stroke technique around some optimal trajectory x_(r)*changes the outcome. The perturbations are expressed in terms of themodel features. Example of features include the initial conditions forthe movement phases such as the forward swing or the characteristics ofthe phase profiles such as the forward swing roll rate. The sensitivityin the outcome relative to these features define the phase transitionderivatives and phase profile derivatives.

The following illustrates these derivatives through the sensitivity ofthe spin outcome to phase transition and phase profile features. Thefour most sensitive spin outcome phase transition derivatives forforehand are:

-   -   (a) azimuth, and    -   (b) elevation at back loop initiation, and the    -   (c) roll rate p, and    -   (d) elevation at the forward swing initiation.

The corresponding derivatives are:

$\begin{matrix}{C_{S,{\Delta AZ_{BL0}}} = \frac{\partial S}{{\partial\Delta}AZ_{BL0}}} & {{EQ}.38}\end{matrix}$ $\begin{matrix}{C_{S,{\Delta EL_{BL0}}} = \frac{\partial S}{{\partial\Delta}EL_{BL0}}} & {{EQ}.39}\end{matrix}$ $\begin{matrix}{C_{S,p_{FSW0}} = \frac{\partial S}{\partial p_{FSW0}}} & {{EQ}.40}\end{matrix}$ $\begin{matrix}{C_{S,{\Delta EL_{FSW0}}} = \frac{\partial S}{{\partial\Delta}EL_{FSW0}}} & {{EQ}.41}\end{matrix}$

Values for all four coefficients can be extracted through linearregression.

The other important characteristics of movement technique that influencethe outcome are the phase profiles. The characteristics are describedthrough phase profiles derivatives. Relevant profiles that change spinimparted to the ball include the peak racket roll rate P_(Fsw) duringthe forward swing or the elevation profile EL_(FSW) during the forwardswing. The respective derivatives are:

$\begin{matrix}{C_{S,{\hat{p}}_{FSW}} = \frac{\partial S}{\partial{\overset{\hat{}}{p}}_{FSW}}} & {{EQ}.42}\end{matrix}$ $\begin{matrix}{C_{S,{\Delta EL_{FSW}}} = \frac{\partial S}{{\partial\Delta}EL_{FSW}}} & {{EQ}.43}\end{matrix}$

FIG. 16D shows the phase profiles for a range of spin outcomes from thebackswing to the impact stroke phases. Inspecting the range of topspinand slice strokes shows clear trends in phase transitions and profilesassociated with these outcomes, allowing the aforementioned derivativesto be visually inferred.

For example, at the start of the backswing, the relationship betweenazimuth and topspin is clear. Strokes that achieve a higher amount ofazimuth rotation generate more topspin. There is similar relationshipbetween elevation and topspin. The strokes with the most negativeelevation generate the most topspin, and the strokes with the mostpositive elevation generate the most slice.

At the start of the forward swing, the phase transition derivatives canalso be appreciated from FIG. 16D. There is again a clear relationshipbetween the elevation angle and the topspin achieved. The roll rate p isnot directly visible on this figure, but examination of the data shows asimilar relationship: a higher negative roll rate leads to more topspinand a higher positive roll rate leads to more slice.

During the forward swing, the relevant profiles can also be examined.Analysis of the roll rate p (not shown on the figure) indicates strongcorrelation between the peak roll rate and the amount of spin impartedto be ball. Generally, the peak roll rate is observed at the end of theforward swing (beginning of the impact), and the peak roll rate at theend of the forward swing corresponds to a unique trajectory. By enteringthis roll rate trajectory and following the trajectory during theforward swing, the user can enter the impact phase with the highest rollrate, leading to more spin on the ball.

There is a similar relationship between the elevation angle profileduring the forward swing and the topspin achieved. While entering theforward swing with the most negative elevation (racket head low) is alsorelated to an increase in topspin, in order to realize that increase, aunique trajectory has to be followed from the start-of-forward swingstate to the impact state. Entering and staying on this trajectory leadsto more topspin, as shown in FIG. 16D.

The outcome phase transition and phase profile derivatives provide thebasis for the design of feedback to help the user maximize theirintended outcomes.

These phase profile and phase transition characteristics can beextracted using a variety of numerical analysis techniques. Theyconstitute the functional skill model that is used for the synthesis ofinstructions and feedback cueing laws.

According to 952 in FIG. 9 the last part of the movement model involvesrelating the phase profile characteristics to movement biomechanics. Thephase profiles are the result of a particular combination of constraintswhich result from body segment, joint angles and muscle activation. Thephase profile characteristics reflect the biomechanics' and otherconstraints such as motor control and task. Movement phases are achievedby exploiting muscle synergies which determine muscle activation andbody segment coordination. Therefore, it is possible to establish arelationship between the biomechanics and extracted strokecharacteristics (see 966 in FIG. 9 ).

In tennis, for example, the general arm motion for the primary strokephases (from forward stroke through the impact to the follow through) isdetermine by the wrist, forearm and shoulder biomechanics. The wrist andforearm represent the ‘end effector’ that provides the fine motorcontrol of the racket, while the gross swing motion is generated throughthe elbow and shoulder, as well as the entire body (torso, hips andlegs).

The forearm pronation and supination (see FIG. 5 ), as well as, wristextension and flexion, are an important component of the finer movementcoordination necessary during the stroke phases needed to generate theoptimal racket swing profiles. For example, the wrist's pronation andsupination is a degree of freedom used for the generating theappropriate impact conditions during the impact phase. In advancedplayers, the arm movement throughout the back loop and transition toforward swing phases, follows very subtle spatial and temporal pattern.

The forearm and wrist motion play a major role in the stroke executionduring the forward swing and impact phase. At the impact the wrist isnominally in neutral position or extended. The amount of extension justbefore impact provides range for flexion that can be used to acceleratethe racket head through a whipping motion. If the flexion at impact istoo large, the generation of racket head acceleration relies entirely onthe elbow, shoulder and body rotation.

With sufficient measurements (e.g. surface EMG electrodes, and bodysegment instrumentation shown in FIG. 5 ) it is possible to map phaseprofiles to joint and body segment motion (which can be described byadditional state variables). Decomposition of the profile and bodysegment variables in synergies provide understanding of how the phaseprofile is achieved and modulated based on musculoskeletal system.

This information can be used to assess the movement profile and phasetransition characteristics taking into account of the individualbiomechanical system. In one implementation, this information canprovide detailed knowledge of strain on the body structure during theexecution of movements and can in turn be used by the cueing system foralerting the player.

The phase analysis at the level of the synergies are used to generatemetrics that describe the compatibility of the movement technique withthe biomechanical system. At the instructions. Stroke structure isdetermined from interactions with the task environment which involveperceptual, decision making and information processing mechanisms.Therefore, measurement of additional quantities such as visual gaze (see12 in FIG. 2 ) provides to encompass perceptual and decision making intothe skill model. The primary decision making and information processingassociated with the stroke are: extraction of cues from the taskenvironment, selection of the stroke (outcome), execution of the stroketo target the outcome, and modulation and adaptation to uncertaintiesand evolving conditions. The primary functional dimensions of the tennisstroke include stroke phases and the states of the oncoming ball(interception, bounce, and impact).

Players select and plan their stroke well before the ball arrives ontheir side of the court based on observations of the opponent and thetactical considerations. The stroke is initiated based on the expectedimpact time and location. As the ball is approaching the player,information about the expected impact conditions can be updated. Theplayer uses the up to date information to modulate the stroke, forexample, the timing of the forward stroke and the orientation of theracket at onset of the forward swing is adjusted during the back loop,and finally, as the ball is within about 150 msec of the impact, theplayer initiates the forward swing. No control is possible during theforward swing therefore the initiation of that phase and the consistencyof the forward swing profile have to be done with great accuracy.

The impact conditions (location on the string bed, ball spin, relativeorientation of the ball relative to the string bed, motion of the racketrelative to the ball) which depend on the conditions of the oncomingball and the player's response and decision about how to engage the ball(see FIG. 2 ), determine the subsequent motion of the ball and theracket. The analysis of the stroke phases in concert with knowledge ofthe perceptual cues used by the player and the knowledge of the othertask elements can be used to assess the decision making and perceptualmechanisms. These assessments can also be used to synthesize cueing lawsto train those mechanisms.

The motion and skill model provide the basis for the synthesis offeedback for the user. The feedback augmentation modalities aredelineated according to two primary categories: instructions that arecommunicated via a communication system and cues that are communicatedvia a cueing system.

The synthesis of instructions and cueing laws follows from the motionmodel 950 as shown in FIG. 9 . Instructions 932 can be derived fromvarious attributes of the skill model; validation cues of outcome andperformance characteristics 934 are primarily movement repertoire andprofile characteristics 962 and 964; and cueing laws 936 are derivedfrom the functional characteristics 966.

Feedback cueing mechanisms are synthesized based on the individual skillassessments and skill model. They are designed to target specificaspects of the movement technique using a variety of feedback signalsthat are generated in real time during the movement.

The example assumes that the user has acquired a range of strokepatterns to achieve a variety of outcomes needed to control the balltrajectory in relationship to the court. These patterns are captured inthe repertoire and assessed following the process described earlier. Theassessment establishes the individual's skill status which specifies thenormal envelope of operation for each pattern, level of outcome andmovement technique variations within that pattern.

The parametric analysis framework described provides the relationshipbetween the key movement features and their effects on the outcome inform of derivatives. The range of outcomes for each movement patternwithin the normal envelope described can be divided into discreteregions according to the outcome level. For example, assuming a normaldistribution in outcome, the mean outcome and the range below onestandard deviation of the mean and the range one standard deviationabove the mean.

The general goals of training or rehabilitation are to improve theconsistency in outcome and expand the level of outcome. Consistency inoutcome requires refinement of the movement technique. Improving theoutcome requires optimizing the movement technique. As discussedearlier, expanding the outcome may require the formation of a new strokepattern that best utilize the biomechanical capabilities. The formationof a new movement pattern is either from the beginning, such as for anunfamiliar movement pattern, or can be achieved from the refinement of aprior movement pattern (see FIG. 15E).

The following list describes the synthesis of instructions. Complexmovements are hard to comprehend due to their high-dimensionality anddynamic nature. Instructions build on the movement structure. Breakingdown the movement in phases helps form an intuitive understanding ofmovement structure and its basic elements.

The assessments are communicated in the forms of reports that provide anoverall summative description of the individual's performance. Thereports combine metrics and visualizations. Assessment componentsorganized by level are:

-   -   Repertoire: description of the movement repertoire highlights        how an individual breaks up the problem space into outcomes.        This is an aspect of skilled behaviors in open skills, which        require users to develop a variety of patterns to address the        broad range of task conditions and outcomes.    -   Repertoire metrics: (i) range of movement patterns and        associated range of outcomes relevant to the domain of        activity; (ii) number of differentiated movement classes; (iii)        coverage of task domain by outcomes; (iv) success rate in        movement class; (v) movement classes with best/worst outcomes.    -   Movement Phases: decomposition of the movement in each class in        terms of movement phases. The goal of assessment at this level        is to help individuals understand the organization and structure        of movement in the context of a task.    -   Movement phase metrics: (i) adaptability of movement to        uncertainties in activity's task conditions and environment.

In addition, the several physical performance metrics can be derivedfrom the phase segmentation and functional analysis including:

-   -   (i) efficiency of the movement technique in achieving their        outcome;    -   (ii) given musculoskeletal constraints, physical stress profiles        at various movement phase.

The following describes the embodied implementation of the visualelements of the User Interface (UI) of the platform. These visualelements enable users to build an understanding of their tennis stroke'smovement structure, and potentially provide an interface to enable orgenerate personalized instructions and real time cues (see FIG. 6 ).This embodiment assumes that the visual elements are deployed via a“mobile app” running on a user's smartphone or smartwatch.

The more complicated interactions with the visual elements will likelytake place during breaks in the training or play. The elements arepresented over a series of “screens” within the mobile app.

The aggregate screen provides a summary of current or past session data,with an emphasis on highlighting the types, durations, & number ofsessions/sets, player info, locations, and types of training performed.The data can be presented as a list or graphically. This screen allowsthe selection of specific sessions/sets/time periods/outcome ranges/etc.These selections can then be used for comparison with other aggregationsor explored on their own, investigating aggregate statistics, trends, orexamining the movements in more detail. The screen allows the creationof a “stroke map” of a user's data based on a specific aggregation.

The system can generate a map of a player's tennis stroke repertoire(so-called “stroke map”) based on the history of all strokes performedwhile using the system was capturing data; this stroke map is displayedon the mapping screen. The stroke map enables understanding of theoverall repertoire versatility and quality of a user's skills. Thisorganization allows for fast understanding of which techniques producewhich outcomes. The mapping enables: mapping of tennis stroke repertoireas the user performs the activity; browsing and navigating the strokerepertoire across sessions of activity; providing outcomes of stroketechnique including ball spin, impact location, and direction; andviewing of stroke technique as stroke paths and associated phasesegments. As will be appreciated by those skilled in the art, thefeatures illustrated in the mapping screen change for non-racquetrelated activities.

The general classification structure for a user's tennis strokerepertoire is the tree shown in FIG. 11 . The tree informs the userinterface interactions relating to exploring the classes. Strokes &their associated information within the different categories 1120 (e.g.,groundstroke, overhead, volley, serve), types 1130 (forehand, backhand),and classes 1140 (FH Class A, FH Class B, FH Class X) can be viewed inmultiple ways. The tree also allows users to navigate down to andbetween individual stroke phases. The stroke 1110 can be segmented intophases 1150 (impact, follow through, recovery, backswing, back loop,forward swing) which has attributes 1160. These plots can be used tohighlight the corresponding trajectory segment along with theperformance and skill attributes derived from the phase features. Theembodied example is based on tennis, however, a user interface to enablethe navigation of movement performance data for other activities wouldinvolve similar general structure, components and modes of interactions.

While the Aggregate Screen was previously described as being used toselect the data for the Mapping Screen, the Mapping Screen itself mayallow selection of data. Options include: current, displaying onlyrecent strokes (recent either by time, session, or set); historicalbests (based on outcomes or other performance metrics).

The Mapping Screen allows for exploration of the classification tree bymeans of specific views. These views are the “Category View”, “ClassView”, “Phase View”, and “Feature View”. The “Category View” (no exampleshown) provides the broadest view across stroke classification types. Itis functionally similar to the “Class View”, covering a broader dataset.The Class View 1910 displays movement categories one level down (in theexample, groundstroke subclasses are shown) showing classes of therepertoire for the appropriate aggregation, organized by movementcharacteristics such as spin (top spin, flat, and slice) and strength(low, medium, and high) (see FIGS. 19A and 19B).

The data can be outlined as a matrix, each cell describes a particularcategory and class. The cells can present basic statistics such as thefraction of different categories and classes (% for the set or session),the number of faulty movements in a class/category (% faults), orstatistics on performances and outcomes. Alternatively, the cells canplot the strokes belonging to the current aggregation, presenting thedata in a more visual format. This includes trajectory ensembles 1912,in the form of time histories (see FIG. 19A), phase diagrams (see FIG.16C-1 to 16C-4 ), 3D movement rotations (see FIG. 16B), histograms,scatter plots, etc. The goal is to provide both intuitive and detaileddescription of the movement patterns. The user can then choose to godeeper in the analysis and select the particular subclass (in tennisspin class and strength) to bring up a “Phase View” (FIG. 19B).

The goal of the Phase View 1920 is to convey detailed information aboutthe racket movement states, movement phases and movement segments for anensemble of strokes or single strokes and their relationship to strokeoutcomes or stroke phase timing. Displaying an ensemble of strokes, byplotting quantities such as swing rate r, roll rate p and elevation θ,for the ‘forward swing’, ‘impact’, and ‘follow through’ phases, providesdetailed descriptions of the phase profiles. Such plots can be viewedeither as a function of time or as a function of racket azimuth (Δφ).

Phase portrait representations (e.g. r vs. ΔAZ), which take away theexplicit temporal characteristics, highlight the particular phaseprofile envelopes for these stroke types (forehand vs. backhand) andstroke classes (topspin, slice). The Phase View can highlight thespecific phases by assigning unique colors; e.g., red for forward swing,black for impact, indigo for follow through. This type of plot makes itpossible to gain detailed insight into the angular motion during thosephases (front view see FIGS. 19A and 19B). The Phase View can alsodisplay similar information as the class view (e.g., 3D movementrotations, histograms, scatter plots, etc.), just applied at the phaselevel.

Phases can also be related to behavioral aspects (Path View, FIG. 18B),biomechanical aspects (Synergy View showing the change in configurationof relevant biomechanical components FIG. 5 over a phase profile), andfunctional factors (Feature View, FIG. 16D). The Feature View FIG. 16Dprovides a detailed representation of the movement features that havebeen identified from sensitivity analysis. The view can selectivelydisplay a set of features that have the strongest impact on the movementoutcome. In the example of FIG. 16B, the view is based on the racketorientation (elevation and azimuth) as viewed on a unit sphere, withorientation relative to the impact datum frame (IDF). Such a plotcombining the coupling between the azimuth and elevation angles isintuitive for understanding the coordination at the various movementphases.

In one configuration, the user can interact by rotating the sphere inFIG. 16B (shown here using two fingers) to inspect relevant aspect suchas the phase transitions. The three panels highlight different views.The “rearview” shows the backswing-backloop-forward-swing transition.The “side view” shows the forward swing-impact-follow throughtransition. The “front view” shows the follow through-recoverytransition. This display highlights features that correspond tovariations in outcomes for strokes. The display also shows the regionsof “normal envelope” that characterizes the individual range of outcomefor this stroke class, and the “extended envelope” that allows theindividual to optimize the outcome given the current movement pattern.

The Feature View helps to teach the user what aspects of the movementtechnique should be emphasized during training to best control the levelof outcome. In the example configuration, the user can interact byrotating the sphere in FIG. 16B (shown here using two fingers) toinspect relevant aspects such as the phase transitions and associatedstates.

The Path View of FIG. 18B provides a detailed representation of themovement technique by visualizing the motion of the racket center ofmass (CM) path in 3D space. A specific movement or stroke can beselected from an ensemble of strokes or from session histories. This 3Drepresentation can be animated to help visualize the racket path orperformance of the movement in each of the phases. The Stroke Path viewcan also be used to depict relevant features of the movement.

Synergy view depicts a relevant body segment during a movement phase.This representation allows the user to understand the implications of amovement technique on the biomechanical system. For example, for theforward swing phase, the synergy view displays the shoulder rotations,elbow extension, forearm pronation and wrist extension (see FIG. 5 )during the phase and highlights potential strains or risks of injurythat have been assessed from the performance. Through a combination ofthe assessment and augmentation features of the system, the system cansuggest injury alert cues that help prevent the body segments performingat-risk movements in one or more phases.

The app supports a number of user interactions with these visualelements. When viewing strokes, the user can pinch and zoom in on asingle stroke or stroke phase segments 1920. Panning and tilting of 3Dvisualizations can also be performed (see FIG. 16B and FIG. 18B).Additional functionalities include tagging and marking stroke phasesegments. A user can also set cues based on the visualized phase segmentfeatures (see FIG. 16B, and FIGS. 19A and 19B), possibly based off aspecific visual prompt, from a selection of relevant cues, or based offa custom user request.

FIG. 24A illustrates cueing mechanisms (outcome validation, alerts, andprofile and transition cues) at different movement phases for tennis.The primary feedback augmentation is derived from the motion model andgeneral movement functional characteristics (see 930 in FIG. 9 ).

Outcome validation: real-time feedback signal to inform player that themovement attained the desired target outcome. For example, the targetoutcome for topspin can be set.

Outcome feature validation: real-time feedback signal to inform playerthat the movement technique for a desired target outcome (see D in FIG.24A). Similar to the outcome validation but in addition, a signal isgenerated to signal movement conformance in terms meeting the featuresfor movement outcome. Potential features include movement state at phasetransition and phase profile.

Outcome feature validations are based on the discretization of theoutcome levels described earlier. Using the same quantization of theoutcome in three tiers as for the outcome validation, for example, amovement technique that produces an outcome at the second level (med)will produce a discrete tone pulse (D) at each the back loop initiationand forward swing initiation (see b) in FIG. 24A). In addition, a pulse(D) is produced at the impact if the outcome reaches the expected mediumspin level (see A in FIG. 24A).

Alerts: real-time feedback cue to signal specific movement features thatare detrimental to performance or outcome. They can also be used toprotect the user from injury. For example, racket backward during thebackswing (see a) in FIG. 24A) or racket elevation at the end of thefollow through (see E in FIG. 24A). The alerts can be used for generalmovement or for a specific movement pattern.

Outcome optimization: real-time feedback cues to help optimize movementtechnique for a specified outcome. The mechanism for outcomeoptimization cueing is derived from the sensitivity analysis. Forexample, the four derivatives that were illustrated for the diagnostic:the racket roll rate and the racket elevation at the forward swinginitiation and the racket elevation and azimuth at the back loopinitiation (see FIG. 16D). The general idea is to provide cues to helpenforce features that are maximize the outcome. This is achieved bycombining feature components based on multivariate derivatives.

The selected phase transition features describe specific aspects of thestroke technique that are correlated with the outcome. For the forehandexample shown in FIG. 16D, when considering the top spin outcome, thesefeatures include the racket elevation and orientation at the back loopinitiation, and forward swing initiation. The result can be implementedas a phase transition or phase profile cue (see B and C, respectively,in FIG. 24A)

An alert cue can be used to help protect the subject from injury or wearcan be implemented based on the analysis of the relationship betweenmovement profiles and biomechanical system. For example, the phaseprofile during the forward swing and impact can be mapped to the wristand forearm joint motion and muscle activity (FIG. 5 ). The synergydecomposition, can be used to determine joint deflections and muscleactivation profiles. This mapping can then be used to detect when thesequantities are exceeding some acceptable values. This information can beused to generate alerts in the form of phase transition and phaseprofile cueing to help form movement patterns that are compatible withan individual's musculoskeletal conditions.

All of these feedback cues help the player develop “muscle memory”through association between the features of the movement phase andcorresponding effects on the outcome. Cueing also helps the playersdevelop understanding of the movement structure and organization.

Note that it is also possible to design more general cue profiles thatare not targeting specific outcome. For example, cues that are usefulfor any forehand or backhand groundstrokes. Such cues emphasize generalaspects of the technique, such as the extend of the backswing, thefollow through. Some features are common to particular stroke classes,therefore, it is possible to have class specific general cue profiles.Such cueing profiles require identifying the stroke class in order toselect the set of cue features appropriate for the class.

Further details of cueing laws are provided as part of the descriptionof the cueing system for the tennis application.

The following describes the system components for the implementation offeedback augmentation. Feedback augmentation is enabled by the cueingsystem.

The cueing system 1000 shown in FIG. 10 has two primary components: acue processor 1010 and a cue generator 1030. The cue processortranslates movement data into cue signals. The cue generator translatescue signals into physical stimuli (in the tennis system, audio soundwaves).

There is no restriction to the layout of the cueing system components,although certain organizations will be more effective or practical. Onesuch implementation has the cueing system operate entirely on theembedded sensing device. Alternatively, the cue processor can operate ona mobile device and the cue generator on a smart watch, with cue signalstransmitted between them. As will be appreciated by those skilled in theart, these different hardware implementations don't affect thefunctional principles.

Within the cue processor, there are two components: the state machine1020 and the cueing law calculator 1012. The state machine is furtherbroken down into a movement phase estimator 1024 and a feature extractor1022.

During operation, preprocessed motion data is sent to the cue processor(see FIG. 10 and operations in FIG. 20B). The preprocessing steptranslates raw sensor data into engineering quantities as well asproviding filtering or physical quantity estimation (i.e., orientation).Quantities relevant to tennis were chosen after analyzing recordedsensor data. The finite state estimator operates as a finite-statemachine. It's defined by the current phase state vector and has rulesgoverning transitions between states. The state vector of the finitestate machine also includes various quantities which are tracked inorder to more effectively estimate the current state. From the provideddata, it estimates the current phase segment of the racquet and player.

A description of a potential implementation of the finite stateestimator follows. As will be appreciated by those skilled in the art,successful phase estimation can be successfully achieved using differentclasses of approaches (finite state machines, HMMs, etc.) or differentparameters for the same approach. This finite state estimator wasdesigned using empirical analysis and is a type of finite state machine.It considers a reduced subset of possible states: ready, backswing,forward swing, impact, and follow through (S=Sr; Sb; St, Si; St). Anexample of its operation is shown in FIG. 20B. The state transitions aregoverned by the following rules:

Ready: the ready state is left when a significant amount of z-axisrotation has been detected, as determined by integrating the z-axisangular rate.

s→S _(b):|∫_(t0c) ^(t) rdt|>r _(∫thresh)  EQ. 44

Or, the absolute value of the integral of the angular rate r between themost recent time where r crossed 0 and now needs to be greater than aspecified threshold. This has the disadvantage of not detecting thestate transition until the middle of the backswing (leading to someinaccuracy), but it does reject a number of “false positive” statetransitions that would otherwise occur. The latency is not an issue forthis transition as cues are typically not delivered on the entry tobackswing.

Backswing: the backswing state transitions to the forward swing statewhen a z-axis rotation rate zero-crossing is detected, or the readystate if excessive time has passed. The forward swing transition isgoverned by the following rule:

s→S _(f): sgn(r(t−))≠sgn(r(t+))  EQ. 45

And the ready state transition is governed by a separate rule:

s→S _(r) :t−t _(0c) >t _(bs,max)  EQ.46

Or, if the time between the start of the backswing and the current timeis greater than the maximum allowable time for a backswing, the machinetransitions to the ready state, as it is assumed something interruptedthe stroke and the impact will not occur.

Forward swing: the forward swing state can transition either into theimpact state or the ready state. The transition to the impact state isdetermined by:

s→S _(i):Σ(|{dot over (H)} _(z) |>{dot over (H)} _(z,threshold) ∧|{dotover (a)}|>{dot over (a)} _(threshold) ∧|a|a _(threshold))=3  EQ. 47

Where {dot over (H)}_(z) is the time derivative of the angular momentumvector's z-axis component. The above rule states that the machinetransitions to the impact state when all three of these statements aretrue during the forward swing: the z component of the time derivative ofthe angular momentum vector is above its threshold, the magnitude of thejerk vector is above its threshold, and the magnitude of theacceleration vector is above its threshold.

While this rule is being checked at each time step, an additional ruleis also being checked:

s→S _(r) : t−t _(0c) >t _(fs,max)  EQ. 48

This is similar to the “backswing exit” rule, which is that if excessivetime has passed, the machine transitions to the ready state.

Impact: the impact state automatically transitions to the follow throughstate when tested:

S→S _(t)  EQ. 49

Follow through: the follow through state will transition to the readystate according to the following rule:

S→S _(r) t−t _(i) >t _(ft,min)∧sgn(r(t−))≠sgn(r(t+))  EQ. 50

Or, the follow through state transitions to the ready state when aminimum amount of time has passed and a zero-crossing for the z-axisrotation rate has occurred after that time passing.

The feature extractor is responsible for combining the phase estimateand the motion data into a feature quantity. As described, features arequantities extracted from the data that have been shown to be relevantto required processing.

For tennis, there are two broad categories of features: continuous anddiscrete. The phase profile features described are a subset of potentialcontinuous features; phase transition features as described are a subsetof potential discrete features. Other discrete features includeoutcomes, alerts, non-transition related timings or actions, or others.

During operation, the feature extractor has a mapping of phases tofeatures-to-compute. For example, during the backswing phase, theextractor has instructions to compute peak azimuth rate, continuous rollrate, elevation at end-of-phase, as well as others. During the impactphase, the extractor's instructions will include computations for theoutcomes.

The relevant features for any given cueing situation will inevitably bea subset of the set of all relevant features for a user, or potentiallyfor all users. However, the set of all relevant features issignificantly smaller than the set of all potential features.Furthermore, as previously discussed, the dimensionality of the humanmovement problem is intractably large. A component which essentiallyonly filters through relevant pieces of information simplifies thesystem design.

The extracted features and the current state estimate are then providedto the cueing law calculator, which is responsible for interpreting themand generating a cue signal. It has its own parameters used to computethe cue signal. The cueing law calculator's parameters are in partdefined by the active cueing profile (as discussed) and by otherparameters relevant to the current user.

The cueing profile is in part responsible for this configuration—theselection (by “THE SYSTEM” or by the user) of a profile will determinewhich cues are currently active. Rather than computing all cues, onlycomputing relevant cues reduces load on the system.

Part of this active configuration is a mapping between phases & featuresto cue signals and quantization levels. For example, such a mappingwould relate starting swing rate (discrete feature) during the forwardswing (phase) to a cue signal for outcome feature validation with levelsat swing rate={0, 5, 10, 15, 20, 25, inf}. Another example is a mappingrelating the elevation rate (continuous feature) during the backswing(phase) to a cue signal for outcome optimization against a targetelevation profile, with levels relative to the target profile at {−40%,−20%, ±5%, 20%, 40%}. An alert cue, for example, can be synthesized fromthe forearm pronation (see FIG. 5 ) to monitor the safe range of motionfor the wrist during the various movement phases. The forearm pronationalert cue maps the relevant quantities to two discrete levels {safe,unsafe}.

Multiple maps across phases, features, signals, and quantization levelscan be active at any given time and multiple signals can be generated.The target or reference values/profiles as in a cueing law 720 (FIG. 7 )will be computed offline, and provided as part of the cueing lawparameters.

The cue signal will be a description of what is being cued on, thelevels, and whether the signal is continuous or discrete. Both thecueing law calculator within the cue processor and the cue encoderwithin the cue generator will need to be configured to have sharedknowledge of what cue signals are currently defined (e.g., if the cueinglaw calculator is sending an outcome cue type signal, the cue encoderneeds to have provisions to handle such a signal).

A more detailed example for the outcome validation cue signal follows.The cue signal can be encoded based on quantized range of outcome, wherequantization levels correspond to outcome quality tiers. Consider a topspin quantization with four tiers S={(200, 500), (500, 1000), (1000,1500), (1500, inf.)} rpm: →{low, med, high, very high}. The cue signalis defined as:

200<|S|<500 rpm→y _(signal)(low)  EQ. 51.1

500≤|S|<1000 rpm→y _(signal)(med)  EQ. 51.2

1000≤|S|<1500 rpm→y _(signal)(high)  EQ. 51.3

1500≤|S|<inf. rpm→y _(signal)(very high)  EQ. 51.4

And, y_(signal) needs to be defined as a “topspin outcome” cue.

These cue signals are used to generate a cue stimulus that can beperceived by the players at the instant of the impact. The cue generatoris responsible for generating this cue stimulus. For tennis, stimuli areaural; either pulses, tones, or continuous waveforms.

As shown on FIG. 10 , the cue generator 1030 is divided into twofunctional blocks: the cue encoder 1032 and the transducer 1034. The cueencoder accepts the cue signal and determines what waveform to provideto the transducer. The transducer then converts the waveform into asoundwave for the user to hear.

As described previously, the encoder takes in cue signals and based offits configuration will generate an audio waveform for the transducer.The cue encoder does not require a sophisticated description of how tointerpret the signals. Its requirements are to have a mapping betweencue signal type, cue signal quantization level, cue signaldiscrete/continuous and corresponding audio waveforms. Rules forcombining mappings are also necessary.

For example, the cue encoder can be configured with a mapping of“outcome feature validation for elevation rate”, “{10, 20, 30, inf.}”,“continuous” to a set of audio waveforms {WF1, WF2, WF3, WF4}. Anotherexample is a mapping of “outcome optimization forward swing featuretransition for swing rate”, “{5, 15, 25, 45}”, “discrete” to a set ofrepeating waveform pulses {WFx1, WFx2, WFx3, WFx4}.

The active mappings, both the definition of the cue signal and theresulting waveforms, are set by the cueing profile, and are determinedby current goals, current user status, historical user status, and anyother relevant information or personal preferences (such as how well agiven user can distinguish between audio notes).

A common “language” between the user and waveform characteristics willneed to be defined, so the user learns to realize certain tones arepositive, or louder signals are negative. Users may have uniquerequirements on the types of cues they can hear, or on how effective agiven cue delivery schema will be for them. For example, some elderlyusers require different frequencies to be used. Some users react morestrongly to volume variations, and others to frequency variations. Allof these waveforms can either be pre-computed and stored in audio files,or generated on-the-fly as needed. For continuous cue signals, aspectssuch as distortion can also encode phase profile variability; if thestroke starts out “rough” the signal may sound awkward or unpleasant,and if the user “smoothens out” the feedback will sound better.

A more detailed example of an encoding schema follows. This encodingmaps outcome tiers to tones on a C major scale (C,D,E):

y _(signal)(low)→y _(stimuli)(low)=pulse(C)  EQ. 52.1

y _(signal)(med)→y _(stimuli)(med)=pulse(D)  EQ.52.2

y _(signai)(high)→y _(stimuli)(high)=pulse(E)  EQ.52.3

y _(signal)(very high)→>y _(stimuli)(very high)=pulse(C+D+E)  EQ. 52.4

An alternate encoding could be mapping to a number of pulses:

y _(signal)(low))→y _(stimuli)(low)=1×pulse  EQ. 53.1

y _(signal)(med)→y _(stimuli)(med)=2×pulse  EQ. 53.2

y _(signal)(high)→y _(stimuli)(high)=3×pulse  EQ. 53.3

y _(signal)(very high)→y _(stimuli)(very high)=4×pulse  EQ. 53.4

The transducer 1034 is responsible for translating the audio waveforminto sound waves for the user to hear. As with any such transducer, anelectrical signal needs to be provided. The audio waveform generated bythe cue encoder may need to be amplified in order for the transducer tohave enough power to generate a loud enough sound.

The encoder might not necessarily output uncompressed time series data;in this case the transducer component would be responsible fortranslating the encoder's output into a directly playable waveform. Thetransducer location may also be relevant on effectiveness. Locating thetransducer on the racquet may provide the user more spatial information,but the signal itself will not be as consistent as if the sounds weregenerated directly in the user's ear (via a headphone for example).

Additionally, the transducer need not be in the same physical locationas the encoder or the rest of the cueing system; e.g., a wirelessspeaker. For the tennis example, the transducer can either be on theracquet, on a mobile device attached to or near the user (e.g.,smartwatch, smart phone on courtside bench), or on a separate device onthe user (e.g., Bluetooth headphones). Finally, the combination of theseand other factors (audio transducer design, audio fidelity, powerrequirements, distance from ear, external noise, etc.) will have animpact on what waveforms should be provided to the transducer. One wayto describe an audio waveform is to characterize its envelope with theparameters: attack, decay, sustain, and release; they describe the timeto peak, time to steady state, steady state/peak ration, and time fromsteady state to zero. The above factors will factor into the choice ofthese parameters.

B. Golf

Motion skills science demonstrates that most movement activities can beanalyzed in terms of primary movement units which are the movementpatterns used for achieving outcomes in a task or activity. The systemis configurable to generalize across many activities and train a widerange of movement types through its ability to extract structure andtrack patterns and from the performance histories of motion data. Thefollowing describes embodiments and implementation examples for thefollowing activities: Golf, Running, Swimming, Skiing, andRehabilitation.

The golf scenario considers a user who wants to improve their golfperformance. Specifically, this includes improving their control overthe golf ball across many different club types, swing types, andsituations. Increased control over the ball will lead to a lower score,improving the user's golf performance.

A minimal implementation of the cueing system for golf consists of asensor with an IMU, attached to the user's club. The sensor needs to beaware of what club is currently in use—accomplished by either having theuser identify which club is being used, or an electronic tag placed onthe club and read by the nearby sensor device. The sensor transmits theIMU data to a mobile phone. The user also wears a smartwatch, andcarries their phone on or near them. The user's mobile phone isresponsible for processing the sensor's and smartwatch's IMU data,calculating cues, and then either delivering audio cues via speaker onthe phone or watch.

A golf session involves the user going to either a driving range,putting green, golf course, or other practice environments. They willconfigure and prepare their equipment. On their phone they will startthe app, creating a session/set/other temporal context. They then mayactivate certain cues or potentially enable training drill (eitheraction possibly being in response to an app prompt).

The following describes the key processes used in the assessment andaugmentation loops (FIG. 1 ). The processes are implemented according tothe processing components described in FIG. 9 . The primary movementunit for golf is the swing.

The devices mentioned above allow to capture user motion data including:club orientation, angular rates, angular accelerations; arm and bodyconfiguration, rates, and accelerations. Relevant environmentalconsiderations are: GPS location, wind velocity, air temperature, and/orcourse (ground) conditions. The outcomes recorded include the quantitiesnecessary to describe ball final position and ball flight path. In oneimplementation, the user's phone is responsible for computing thesequantities; in another implementation, outcome data is acquired viaalternate means.

As is the case with other open motor skills there are multiple primarymovement patterns; for golf, these are different swing types. For golf,the swing type is primary determined by club selection (which isdirectly related to the user's current goals). Each club-dependent swingtype will also have multiple classifications, based off other intendedoutcomes.

Each classification will have a unique phase segmentation as well. FIG.4B shows examples of how the phase segmentation will vary based off thetype of swing. Some swings will have a backswing, a 2-phase downswing,impact, follow through, and recovery; other swings will have only abackswing, downswing, impact, and upswing. Phase segmentation can beaccomplished using a similar approach to what was described for thetennis example.

For golf, the representation of skill is heavily determined by thebreadth of the user's repertoire that displays a reasonable amount ofperformance. Playing a game of golf involves many long shots, shortshots, puts, etc. Having no swing or an incredibly poor swing in onearea can easily drive up a user's score.

The skill model also encompasses the information described elsewhere inthis document, recording: relationships between movement techniques andoutcomes; outcome, repertoire, and/or technique trends; effectiveness ofcues; and other quantities.

For golf, any of the previously mentioned cueing modalities could beused. Cues include audio feedback from either the user's phone orsmartwatch. As golf is a relatively intermittent activity, there is moreflexibility in how the information can be encoded. Likely choices aretone (audio note), duration, or repetition; more complicated coding(chords, chord progressions), visual feedback, or verbal feedback arealso possible.

Example cues are shown in FIG. 24C and include:

-   -   Phase Profile Cue—Backswing Angular Rate. To either improve or        reinforce a technique, this cues the user on an appropriate        angular rate profile.    -   Alert Cue—Max Backswing Rotation. Raising the club too far back        can cause injury; this cue helps the user avoid this action.    -   Phase Profile Cue—Peak Acceleration. In order to maximize        distance, the user needs to accelerate their club during the        swing to match a target acceleration profile; this cue assists        with this.    -   Outcome Validation Cue—Target Outcome. This cue informs the user        of the outcome they just achieved when hitting the ball (e.g.,        strength of shot, angle, etc.).    -   Phase Profile Cue—Follow through extension. Impacting the ball        correctly requires the club to be swinging relatively fast.        Cueing the user on their follow through helps to build        association.

Combinations of cues can be delivered to support technique and outcomedevelopment. One potential scenario has primary and secondary cuescombined to support a more efficient golf swing trajectory whilemanaging injury risk. An example cueing profile might be: primary cue:Alert Cue—Max Backswing Rotation; secondary cue: Outcome ValidationCue—Target Outcome. As will be appreciated by those skilled in the art,cue profiles can be made of many combinations of cues and the selectionof the cue profile will have many factors.

C. Running

The running scenario considers a competitive endurance runner trainingto win a half marathon type event. The general outcome goal is toincrease stride efficiency. Increasing stride efficiency will allow auser to move faster for the same effort, thereby increasing averagespeed over the course, and reducing time. A minimal implementation ofthe cueing system for running consists of a sensor attached to a user'sfoot and a phone or smartwatch attached to a user's arm or body which isin communication with the foot sensor. Both the sensor and the host inthis system record movement data from an IMU. The phone or smartwatchcapture GPS data. Processing is run on the host, and cues are deliveredby the host via audio feedback. A running session involves a user makinga decision about the goal of the session. These include training withprompted drills, entering specific routines, or just running without aspecific goal.

The following describes the key processes used in the assessment andaugmentation loops (FIG. 1 ). The processes are implemented according tothe processing components described in FIG. 9 . The primary movementunit of running is one stride of the gait cycle.

The devices mentioned above allow to capture user motion data including:body configuration and speed (angles, CG height, velocity); leg segmentsconfiguration and speed (hip/knee/ankle angles and rates). The outcomesof interest are: efficiency (Energy/Stride Unit/Distance/Stride Unit);maximum possible speed; forces on joints; muscle type usage distribution(division of force between slow & fast muscle fiber types). Theseoutcomes can be computed using a gait motion estimator. Thesecalculations are performed on the user's host device.

As is the case with other open motor skills the repertoire includesdifferent type of movement patterns. For running, these are differentgait cycles. FIG. 4E shows some of these, including walking, jogging,running, and sprinting, as well as their more detailed subclasses,determined by other criteria such as slope or target speed. Each gaitcycle class has a unique phase segmentation.

FIG. 24D illustrates cueing mechanisms for running. For the fast runninglevel ground gait cycle the phase segmentation is Leap, Impact, Drive,Recovery (FIG. 24D). The phase segments can be identified using existinggait models. The phase segmentation is specific to the gait classes.

The skill model for running, includes the repertoire of gait types(e.g., walking, jogging, running, sprinting) as well as a more detailedclass breakdown. For this example, the more important aspect of skill isthe performance and not the repertoire breadth. The skill model alsoencompasses the information described elsewhere in this document,recording: relationships between movement techniques and outcomes;outcome, repertoire, and/or technique trends; effectiveness of cues; andother quantities.

The cueing modalities for running include any of the previouslymentioned cueing modalities. Audible cues can be generated from eitherthe user's phone, smartwatch, or either via headphones. The informationcan be encoded with characteristics such as tone (audio note), duration,or repetition.

Example cues are shown in FIG. 24D and include:

-   -   Alert Cue—Foot height alert; lifting a foot too high during        recovery wastes energy and therefore reduces efficiency.    -   Profile Cue—Foot swing trajectory; having proper coordination of        leg segment movements will lead to more efficient, less        joint-impact strides. By cueing towards a profile for the foot        to follow during the leap phase, target values for leg segment        velocity and acceleration can be achieved.    -   Alert—Impulse limit on ground contact cues runner on how hard        their feet are hitting the ground on impact.    -   Phase Feature Cue—Drive extension; the duration of the        foot-ground contact and the leg extension during the contact        period both affect the stride efficiency.

The cue profile provides a combinations of cues that can be delivered tosupport technique and outcome development. One potential scenario hasprimary and secondary cues combined to both maximize efficiency. Anexample cueing profile might be: primary cue: Profile Cue—Foot SwingTrajectory; secondary cue: Phase Feature Cue—Drive Extension. As will beappreciated by those skilled in the art, cue profiles could be made ofany combination of cues and the selection of the cue profile will havemany factors.

D. Swimming

The swimming scenario considers a competitive swimmer who is working ontechnique for speed and endurance. Specifically, the user wants toincrease the efficiency of their freestyle stroke. A minimalimplementation of the cueing system for swimming consists of: a heartrate monitor worn on the chest with an integrated IMU; instrumentedgloves worn on the hands containing multiple pressure sensors and IMU;an earpiece which communicates audio cues. Either the chest IMU or theearpiece is responsible for determining position in the pool, via IMUsensors or possibly a connection to the phone (camera, radio strength,other). The phone receives data from all the sensors, process the data,calculate cues, and communicate the audio cues to the earpiece. Aswimming session involves a user making a decision about the goal of thesession, potentially train with prompted drills, entering specificroutines, or just swimming without a specific goal. The training drillsmight be focused on improving an aspect of skill as determined by theuser, the system, or another individual (i.e., coach).

The following describes the key processes used in the assessment andaugmentation loops (FIG. 1 ). The processes are implemented according tothe processing components described in FIG. 9 . The primary movementunit of swimming is the stroke cycle shown in FIG. 4F. The devicesallows capturing user motion data including: wrist/elbow/shoulderangles; position in pool; body roll; pressure on hand (force & sideslipof end-effector). Outcomes of interest are stroke length, strokeduration, and stroke efficiency. The outcomes can be computed fromkinematic and dynamic models and sensor fusion algorithms on the user'shost device.

As is the case with other open motor skills, the repertoire includesdifferent type of movement patterns. For swimming, these are thedifferent stroke types, such as freestyle/crawl, butterfly, backstroke,etc. There are also variations in speed for each of these types (e.g.,sprint, endurance). Each class has a specific phase segmentation.

For the freestyle stroke, the phase segments are: pull, recovery, reach,and catch, as shown in FIG. 4F. The pull is the propulsive segment ofthe motion. The recovery and reach are the phases that lead to the nextpull, and the catch corresponds to the initiation of the next pull.During the pull, the orientation of the wrist relative the forearm isimportant, as well as the orientation of the forearm in the water.Having the hand & forearm perpendicular to the direction of motionimproves the stroke efficiency.

Stroke length is in part determined by how far the hand reaches forwardduring the reach, as well as how far back the hand/forearm move duringthe pull. Body roll is periodic with the stroke; an appropriate amountof body roll will result in an improved stroke efficiency. Users need tobe careful with shoulder movement during the recovery/reach transition.The phase segments can be identified by the shoulder rotation angle, andthe amount of force being generated by the hand/forearm (via thepressure sensors) can also contribute. Different classes of movement mayhave entirely unique phases and phase segmentation rules though.

For swimming, the representation of the skill encompasses the repertoireof “gait” types (e.g., freestyle/crawl, butterfly, breaststroke,backstroke, etc.), as well as, a more detailed class breakdown similarto the tennis stroke class tree in FIG. 11 . While the user's goal maybe to improve their freestyle stroke, repertoire broadness is stillvaluable for example when competing in medley events. The skill modelalso encompasses the information described elsewhere in this document,recording: relationships between movement techniques and outcomes;outcome, repertoire, and/or technique trends; effectiveness of cues; andother quantities.

The cueing modalities for swimming include the same ones as any of thepreviously mentioned cueing modalities. Cues are audio feedback providedvia the user's earpiece. The information can be encoded withcharacteristics such as tone (audio note), duration, or repetition.

Example cues are shown in FIG. 24E and include:

-   -   Alert Cue—Shoulder Rotation. Shoulder rotation injury/damage is        possible during the recovery phase. Putting an alert limiting        shoulder extension during this phase can therefore help prevent        injuries. The “high risk” zone are identified from the users'        biomechanical properties. Sensor fusion techniques are used to        compute the current configuration of the arms, and detect if the        shoulder extension is in the high-risk zone.    -   Phase Profile Cue—Body Roll. This cue is used to assist the user        maintain target body roll throughout the stroke motion, thereby        helping maximize stroke efficiency.    -   Phase Profile Cue—Wrist/Forearm Orientation. Hand/forearm        orientation during the stroke is a continuous quantity that        affects stroke efficiency that can be cued during the pull phase        (phase profile cue).    -   Outcome Validation Cue—Stroke Length. Validation cue for the arm        extension at the catch transition to help maximize stroke        length.

The cue profile combines cues to support technique and outcomedevelopment. One potential scenario has primary and secondary cuescombined to both maximize stroke efficiency, and a separate alert cue toprevent injury. An example cueing profile might be: primary cue: PhaseProfile Cue—Body Roll; secondary cue: Wrist/Forearm Orientation; alertcue: Alert Cue—Shoulder Rotation. As will be appreciated by thoseskilled in the art, cue profiles can be made of any combination of cuesand the selection of the cue profile will have many factors.

E. Skiing

The skiing scenario considers an amateur skier who wants to improvetheir control during carving, of turn-in and turn coordination. Aminimal implementation of the cueing system for skiing consists of: asensor with an IMU attached to the ski-boot; the user's mobile phonecapturing IMU and GPS location data. The phone receives data from allthe sensors, process the data, calculate cues, and communicate the audiocues to headphones or an earpiece.

The following describes the key processes used in the assessment andaugmentation loops (FIG. 1 ). The processes are implemented according tothe processing components described in FIG. 9 . The primary movementunit of skiing considered here is the carve turn segment (see FIG. 4D).The devices mentioned above allow to capture user motion data including:body, ski, and leg orientations and speeds. The outcomes of interestare: turn rate, side slip and level of turn coordination. These outcomesare computed from the available data; these calculations can beperformed on the user's host device.

As is the case with other open motor skills, the repertoire includesdifferent type of movement patterns. For skiing, these are differentmaneuvers used to negotiate the slope environment, such as followingslaloms, carving, or crossing moguls. Each maneuver type and associatedmotion unit class has its specific phase segmentation. For carving, thephase segmentation consists of: Entry, Load, Apex, Unload (see FIG. 4D).The phase segments can be primarily identified by body and ski roll andyaw rates, as extracted from the sensor data.

For skiing, the skill assessment encompasses the repertoire of maneuvertypes (e.g., carving, moguls). In this scenario focusing on carvingperformance, the repertoire describes the carving maneuvers over a rangeof speeds and turn radii. The skill model also encompasses theinformation described elsewhere in this document, including:relationships between movement techniques and outcomes; outcome,repertoire, and/or technique trends; effectiveness of cues; and otherquantities.

The cueing modalities for skiing, include any of the previouslymentioned cueing modalities. Cues are audio feedback delivered from theuser's phone via headphones. The information can be encoded withcharacteristics such as tone (audio note), duration, or repetition.

Example feedback cues are shown in FIG. 24B and include:

-   -   Phase Profile Cue—Body roll rate profile as the skier enters the        turn.    -   Phase Profile Cue—Body roll and yaw rate profile to train proper        turn coordination. During loading the skis going into a turn,        the body roll angle, ski yaw rate, velocity, and side slip have        to be coordinated.    -   Outcome Validation—Turn rate target to provide feedback about        turning performance.    -   Alert Cue—Side slip warning to prevent loss of energy during the        turn.

Cue profile combine cues to support technique and outcome development.One potential scenario has primary and secondary cues combined to reachtarget side slip values for different turn radii. An example cueingprofile might be: primary cue: Phase Profile Cue—Body Roll and TurnCoordination; secondary cue: Phase Profile Cue—Body Roll. As will beappreciated by those skilled in the art, cue profiles can be of anycombination of cues and the selection of the cue profile will have manyfactors.

F. Rehabilitation

The rehabilitation scenario considers a user who suffered from a stroke,is paralyzed on one side, and is learning to regain the use of theirright hand and arm. A minimal implementation of the cueing system forrehabilitation can consist of: three combined IMU/EMG wireless sensorsworn on the back of the right hand, the right wrist, and the right upperarm; a mobile phone. The phone receives data from all the sensors,process the data, calculate cues, and output the audio cues.

The following describes the key processes used in the assessment andaugmentation loops (FIG. 1 ). The processes are implemented according tothe processing components described in FIG. 9 . While the entirerehabilitation process would involve many different types of movementunits, the primary movement unit considered in this example is the armmotion for reaching, grasping and lifting the arm and hand.

The devices mentioned above allows capturing the user motion dataincluding: motion of the right arm segments and hand (path andorientation) as well as the relevant muscle activity for these bodysegments. Outcomes of interest are duration, quality of movement(accuracy, stability).

The primary movement units considered here are segments of arm motionthat are characterized by parabolic speed profiles, starting from restand ending at rest. This allows phases to be defined as thesub-movements in this process. The trajectories are first segmentedaccording to the movement units in the task activity. This process usesthe parameters determined during the baseline assessment sessions (whichthen will likely change over time). For the rehabilitation case, furtherdecomposition of movement and submovement phases into muscle synergiesis performed via the sensor's EMGs. This analysis allows analyzing howneuro-motor and/or physical strength are recovering.

The cueing modalities for rehabilitation can include any of thepreviously mentioned ones. Cues can be audio feedback from the user'sphone. The information can be encoded with characteristics such as tone(audio note), duration, or repetition. Since rehabilitation movementsunfold more slowly and are more intermittent, there is more flexibilityin how the information can be encoded. In some implementations, visualor verbal feedback can be used.

Example cues are shown in FIG. 24F and include:

-   -   Profile Cue—Elbow/Shoulder Coordination. When reaching for an        object, a certain amount of combined shoulder movement and        corresponding elbow movement is expected. To help the patient        achieve this coordination, cues can be provided to keep them on        the correct profile.    -   Alert Cue—Elbow Excursion. As part of rebuilding correct        movements, limiting the total amount of variation or excursion        from a nominal trajectory is beneficial. An alert cue could warn        these situations to the user.    -   Outcome Validation—Time & Accuracy. Regaining functional        movement performance includes being able to successfully        accomplish tasks in reasonable times. Giving outcome feedback on        each action will be beneficial for a user.

Combinations of Cues can be delivered to support technique and outcomedevelopment. For rehabilitation, user capabilities may varysignificantly over time. Cue profiles may be limited to only a primarycue at first, and over time may integrate all the example cues discussedabove. Analysis of the performances will strongly impact the availablecue profiles.

Before the augmented rehabilitation sessions can be performed, thepatient performs several baseline assessment sessions with a physicianor physical therapist. These sessions are used to establish initialmotion and skill models, baseline statuses, and realistic targets fortraining schedule based on clinical data. Depending on the user andconditions, these setup sessions include vision-based movement trackingsystems to calibrate various algorithms, especially those used in theestimation of the body segments and muscle activations. Therehabilitation process is then organized in half-hour sessions to becompleted on a daily basis. Each session can be divided into setsfocusing on specific training elements. The training elements aregenerated from the results of the assessment, overall training goals, orother potentially relevant clinical data.

The training sessions are composed of training elements that emphasizedifferent aspect of the movement rehabilitation. Training elements arereaching exercises. These exercises involve moving the hand betweenpoints on horizontal plane at standard table height in front of thesubject, and from there to various points on the face including mouth,eyes and nose. The points are designated on a custom tablecloth andauditory cues are used to direct the subject during the session,including the designation of the points on the tablecloth, the timing ofthese movements and cueing mechanisms during the reaching movement tohelp refine the movement patterns. All the training elements areperformed “empty handed” or include weights or accessories such as aglass or fork. The accessories are used to change movement conditions.

Following the training session, the system is used to monitor thesubject's movement in natural environment interactions, for examplewhile preparing their meal or during their meal. The collected data canalso be incorporated by the system to plan the subsequent trainingsessions. Typical updates in training elements include: Introduction ofnew reaching patterns to increase the movement repertoire. Movementrefinement/optimization exercises to improve outcomes (movementprecision, duration, repeatability). Feedback cues are used to targetdeficiencies in the movement coordination and execution.

In particular, to retrain patterns of muscle activation associated withthe muscle synergies used in reaching movements. Timing can be changedto increase the speed of the movement patterns and sensory-motorconsolidation. New movement patterns can be introduced to addressdeficiencies in the repertoire. The results from the daily session arealso relayed to the physician/therapist who designs the long-termtraining goals based on clinical data and skill development data frompatient with similar skill status and clinical presentations.

While preferred embodiments of the present invention have been shown anddescribed herein, it will be obvious to those skilled in the art thatsuch embodiments are provided by way of example only. Numerousvariations, changes, and substitutions will now occur to those skilledin the art without departing from the invention. It should be understoodthat various alternatives to the embodiments of the invention describedherein may be employed in practicing the invention. It is intended thatthe following claims define the scope of the invention and that methodsand structures within the scope of these claims and their equivalents becovered thereby.

1. A user interface system comprising: a processor in communication withone or more sensors configured to obtain motion data from a subjectperforming a task or activity in an environment, the processorconfigured to: receive the motion data from the one or more sensors;parse the received motion data into movement units of the subject and arange of outcomes associated with the movement units; delineate theparsed motion data into movement classes wherein a repertoire of themovement classes is formed; analyze one or more of the movement classesin the repertoire, wherein feedback is synthesized based on anassessment of how the outcomes and conditions of the one or moreanalyzed movement classes in the repertoire compare to the outcomes andconditions required to accomplish the task or activity, and on one orboth of reliability and efficiency of the one or more analyzed movementclasses in achieving one or more of the range of outcomes; and identifyone or more specific aspects of the parsed motion data for change basedon the comparison, wherein the processor relates the identified aspectsto variations in the range of outcomes and wherein one or more of saididentified aspects are predictive of a desired outcome in the range; anda user interface configured to generate the feedback via a user device,the feedback comprising visual, audio, verbal or haptic signals selectedto target the one or more aspects of the parsed motion data identifiedas predictive for the subject to achieve or improve the desired outcomein the range. 2-10. (canceled)
 11. The user interface system of claim 1,further comprising a transducer configured to provide the feedback tothe subject during or after performance of the task or activity, whereinthe feedback comprises an audio stimulus, verbal instruction or hapticsignal generated via the transducer.
 12. The user interface system ofclaim 1, wherein the user device is selected from a smart phone, a smartwatch, smart glasses, an earpiece, a headphone or headphones, aBluetooth device, and a tablet computer or mobile computer. 13.(canceled)
 14. The user interface system of claim 1, wherein one or moreof the sensors are provided or mounted in or on equipment used by thesubject in performing the task or activity, or worn by the subject whileperforming the task or activity. 15-17. (canceled)
 18. The userinterface system of claim 1, wherein the feedback comprises a repertoiremap displayed on the user device, wherein the repertoire map describesone or more of the movement classes or subclasses of the movementclasses.
 19. The user interface system of claim 1, wherein the userdevice is adapted for selecting one or more of the movement classes orsubclasses to display; and wherein the movement classes are divided intothe one or more subclasses based on relative similarity of therespective movement units; and/or wherein the movement classes aredelineated into the one or more subclasses based on the outcomesassociated with the respective movement units.
 20. (canceled)
 21. A userinterface comprising: an input for collecting motion data from one ormore sensors, wherein the one or more sensors are configured to obtainthe motion data for a subject performing a task or activity in anenvironment; wherein the collected motion data are parsed into movementunits and a range of outcomes associated with the movement units, andcompared to a repertoire formed by delineating the movement units intomovement classes used by the subject to accomplish the range of outcomesin the task or activity; and wherein the processor is configured to:generate a movement phase estimation which provides a prediction of amovement phase and an associated movement feature that relates tovariation in the range of outcomes; extract the associated movementfeature, wherein the associated movement feature is identified aspredictive of a desired outcome in the range; and generate feedback viaa user device based one or both of reliability and efficiency of one ormore of the movement classes in achieving one or more of the range ofoutcomes, wherein the feedback is generated in visual, audio, verbal orhaptic form selected to target the associated movement featureidentified as predictive for the subject to achieve or improve thedesired outcome in the range.
 22. (canceled)
 23. The user interface ofclaim 21, wherein the user interface is presented on a smart phonedisplay, a smart watch display, smart glasses, an earpiece, a headphoneor headphones, a Bluetooth device, or a tablet computer or mobile deviceor mobile computer display.
 24. The user interface of claim 21, whereinthe feedback is provided to the subject in audio, verbal or haptic formduring performance of the task or activity or in audio, verbal orgraphic form after the performance of the task or activity.
 25. The userinterface of claim 21, wherein one or more of the sensors is provided ormounted in or on equipment controlled by the subject or worn by thesubject in performing the task or activity, and wherein the processor isconfigured for communication with the one or more sensors via a wired orwireless connection.
 26. (canceled)
 27. The user interface of claim 21,wherein the feedback comprises a repertoire map displayed on the userdevice, wherein the repertoire map describes one or more of the movementclasses or subclasses of the movement classes, and wherein the userdevice is adapted for selecting one or more of the movement classes orsubclasses of the movement classes for display.
 28. A method comprising:collecting motion data from one or more sensors, the motion dataresponsive to a subject performing a task or activity in an environment;parsing the collected motion data into movement units and a range ofoutcomes associated with the movement units wherein a repertoire ofprior movement units is formed by delineating the movement units intomovement classes used by the subject to accomplish the range of outcomesin the task or activity; analyzing one or more of the modeled movementclasses; generating an assessment of how the outcomes and conditions ofthe one or more analyzed movement units classes in the repertoirecompare to the outcomes and conditions required to accomplish the taskor activity, and on one or both of reliability and efficiency of the oneor more analyzed movement classes in achieving one or more of the rangeof outcomes; identifying one or more specific aspects of the parsedmotion data for change, based on the comparison, wherein he identifiedaspects of the parsed motion data relate to variations in the range ofoutcomes and one or more of the identified aspects are predictive of adesired outcome in the range; and providing feedback to a user device,the feedback selected from visual, audio, verbal or haptic signalstargeting the one or more aspects of the parsed motion data identifiedas predictive for the subject to achieve or improve the desired outcomein the range.
 29. The method of claim 28, wherein the user device isselected from a smart phone, a smart watch, smart glasses, an earpiece,a headphone or headphones, a Bluetooth device, and a tablet computer ormobile computing device. 30-34. (canceled)
 35. A non-transitory,computer readable data storage medium having program code storedthereon, the program code configured for execution on a processor of amobile computing device to perform a method according to claim 28.36-37. (canceled)
 38. The method of claim 28, wherein the feedbackcomprises a repertoire map describing one or more of the movementclasses, and wherein the repertoire map is configured for selecting oneor more of the movement classes or subclasses of the movement classesfor display on the user device.
 39. (canceled)
 40. The user interfacesystem of claim 1, wherein the feedback comprises instructions selectedto improve one or more of the range of outcomes, or to impact thereliability or efficiency of one or more of the movement classes inachieving one or more of the range of outcomes.
 41. The user interfaceof claim 40, wherein the instructions are generated during performanceof the task or activity by the subject, following one of movement unitsof the subject, or after a training session of the subject.
 42. The userinterface system of claim 18, wherein the repertoire map comprises amatrix with a plurality of cells, each of the cells describing one ofthe movement classes, or a subclass of one of the movement classes. 43.The user interface system of claim 42, wherein the plurality of cellsare organized by movement characteristics of the one or more movementclasses or subclasses, the movement characteristics including one ormore of stroke type, spin and strength.
 44. The user interface system ofclaim 42, wherein the repertoire map is adapted for selection of astatistic or trend related to one or more of the movement classes orsubclasses.
 45. The user interface system of claim 19, wherein the userdevice is adapted to display one or more attributes of the one or moreselected movement classes or subclasses.
 46. The user interface systemof claim 45, wherein the one or more attributes are selected frommovement features associated with one or more of the outcomes, phaseprofile characteristics, stroke categories, or stroke features.
 47. Theuser interface of claim 21, wherein the feedback comprises instructionsselected to impact the reliability or efficiency of one or more of themovement classes in achieving one or more of the range of outcomes,wherein the instructions are generated during performance of the task oractivity by the subject, following one of movement units of the subject,or after a training session of the subject.
 48. The user interface ofclaim 27, wherein the repertoire map is adapted for selecting astatistic or trend related to one or more of the selected movementclasses or subclasses.
 49. The user interface of claim 27, wherein therepertoire map is adapted for displaying one or more attributes of theone or more selected movement classes or subclasses.